NASA-TM-2017-219598 


Unmanned Aircraft Systems Minimum Operational 
Performance Standards End-to-End Verification and 
Validation (E2-V2) Simulation 


Rania W. Ghatas 
Langley Research Center, Hampton, Virginia 


Devin P. Jack 
Adaptive Aerospace Group Inc., Hampton, Virginia 


Dimitrios Tsakpinis 
Science Applications International Corporation, Hampton, Virginia 


Michael J. Vincent 
Langley Research Center, Hampton, Virginia 


James L. Sturdy 
Stinger Ghaffarian Technologies Inc., Hampton, Viriginia 


Cesar A. Munoz 
Langley Research Center, Hampton, Virgnia 


Keith D. Hoffler 
Adaptive Aerospace Group Inc., Hampton, Virginia 


Aaron M. Dutle 
Langley Research Center, Hampton, Virginia 


Robert R. Myer 
Stinger Ghaffarian Technologies, Inc., Hampton, Virginia 


Anna M. DeHaven 
Craig Technologies, Inc., Hampton, Virginia 


Elliot T. Lewis and Keith E. Arthur 
Langley Research Center, Hampton, Virginia 


April 2017 


NASA STI Program .. . in Profile 


Since its founding, NASA has been dedicated to the 
advancement of aeronautics and space science. The 
NASA scientific and technical information (STI) 
program plays a key part in helping NASA maintain 
this important role. 


The NASA STI program operates under the auspices 
of the Agency Chief Information Officer. It collects, 
organizes, provides for archiving, and disseminates 


NASA’s STI. The NASA STI program provides access 


to the NTRS Registered and its public interface, the 
NASA Technical Reports Server, thus providing one 
of the largest collections of aeronautical and space 


science ST] in the world. Results are published in both 


non-NASA channels and by NASA in the NASA STI 
Report Series, which includes the following report 


types: 


e TECHNICAL PUBLICATION. Reports of 


completed research or a major significant phase of 


research that present the results of NASA 


Programs and include extensive data or theoretical 


analysis. Includes compilations of significant 
scientific and technical data and information 
deemed to be of continuing reference value. 
NASA counter-part of peer-reviewed formal 
professional papers but has less stringent 
limitations on manuscript length and extent of 
graphic presentations. 


e TECHNICAL MEMORANDUM. 
Scientific and technical findings that are 
preliminary or of specialized interest, 
e.g., quick release reports, working 
papers, and bibliographies that contain minimal 
annotation. Does not contain extensive analysis. 


e CONTRACTOR REPORT. Scientific and 
technical findings by NASA-sponsored 
contractors and grantees. 


e CONFERENCE PUBLICATION. 
Collected papers from scientific and technical 
conferences, symposia, seminars, or other 
meetings sponsored or co-sponsored by NASA. 


e SPECIAL PUBLICATION. Scientific, 
technical, or historical information from NASA 
programs, projects, and missions, often 
concemed with subjects having substantial 
public interest. 


e TECHNICAL TRANSLATION. 
English-language translations of foreign 
scientific and technical material pertinent to 
NASA’s mission. 


Specialized services also include organizing 
and publishing research results, distributing 
specialized research announcements and feeds, 
providing information desk and personal search 
support, and enabling data exchange services. 


For more information about the NASA STI program, 
see the following: 


e Access the NASA STI program home page at 
http://www.sti.nasa.gov 


e E-mail your question to help@sti.nasa.gov 


e Phone the NASA STI Information Desk at 
757-864-9658 


e §=6Write to: 
NASA STI Information Desk 
Mail Stop 148 
NASA Langley Research Center 
Hampton, VA 23681-2199 


NASA/TM-2017-219598 


Unmanned Aircraft Systems Minimum Operational 
Performance Standards End-to-End Verification and 
Validation (E2-V2) Simulation 


Rania W. Ghatas 
Langley Research Center, Hampton, Virginia 


Devin P. Jack 
Adaptive Aerospace Group Inc., Hampton, Virginia 


Dimitrios Tsakpinis 
Science Applications International Corporation, Hampton, Virginia 


Michael J. Vincent 
Langley Research Center, Hampton, Virginia 


James L. Sturdy 
Stinger Ghaffarian Technologies Inc., Hampton, Virginia 


Cesar A. Munoz 
Langley Research Center, Hampton, Virginia 


Keith D. Hoffler 
Adaptive Aerospace Group Inc., Hampton, Virginia 


Aaron M. Dutle 
Langley Research Center, Hampton, Virginia 


Robert R. Myer 
Stinger Ghaffarian Technologies Inc., Hampton, Virginia 


Anna M. DeHaven 
Craig Technologies Inc., Hampton, Virginia 


Elliot T. Lewis and Keith E. Arthur 
Langley Research Center, Hampton, Virginia 


National Aeronautics and 
Space Administration 


Langley Research Center 
Hampton, Virginia 23681-2199 


April 2017 


The use of trademarks or names of manufacturers in this report is for accurate reporting and does not 
constitute an official endorsement, either expressed or implied, of such products or manufacturers by the 
National Aeronautics and Space Administration. 


Available from: 


NASA STI Program / Mail Stop 148 
NASA Langley Research Center 
Hampton, VA 23681-2199 
Fax: 757-864-6500 


Table of Contents 


TABLE OF TABLES icdccsiciccciceccdedsistsadiscccecdecsisivecavecccedeledh sdasaveccdediiaiscena dec cedsssiaisasiveddeedésaiasibacesteasiadacsaeces Hl 
TABLE OF ‘FIGURES ceciesssccssscvccedcieacesicdsscecctacnslassstelvedeteanssadasinedelesnansdoouclesdsdedasasdoeselvcdsTssnsassavecetacneascabece Hl 
LIST OF ACRONYMS iisccccssavccdiscteccdcxcsacccdescuckcessescccdsssaccdcessasccctscsecceasshacelssSencceasseuscddbclacadasesatesdasctecstasen Vil 
ABSTRACT issccseescesace ScvevieusdedesutccesccstadsasseTasssses'sdedesecsicecdsedcdesade tsbesdéssdidedads Susevienadedasecbacesidsdidessestsesecetede 1 
INTRODUCTION vicciiccctivctesaccedicnccescaaccececdacesvteas cccvetecaceuecadacedccdstechessedsceddeve cochveds cdcceddencsusssadceeceuacauacssnees 2 
4s) --GENERAL INFORMATION iiicccccic sacccccccicdecassanccec dccecescestvccescccccadsteanccesaccaseasenadcecdscesesdaseveccstecccses seeds 4 
1.1 APPROACH AND OBJECTIVE.....scccccssscesssscsssecssstecssccucossesssaeseseecsnsecseseesssesseaecssauecseseessaseseaaeseacecsensesseess 4 

2: SIMULATION ENVIRONMENT sicsssiccedscicsesesdaccvtecsdansectedccsst oclecessslacbetictivcscvedsbesbosstecsscscbocsbcsiseeceetens 4 
2.1 E2-V2 ARCHITECTURE d22.t0ee2220 ieee eked hailed eats Sadana ete iad 4 
2.2 AIRCRAFT: FLIGHT DYNAMICS t02..20ss8.svebteviocetee Hesse kecbtd deine nde van eee si ees 5 
2.3 SENSOR/ TRACKER SUITE 3: ovcsduvs sec desta sees fevabeed i fesddede ated vasvesdevicgsiedvieiativebest Woessddeieeveda eaten eaveas 6 
2.4 SENSOR UNCERTAINTY MITIGATION APPROACH .......ccesscessscesssecseseeseeecsueeessuecseseeceeeeeseueesaeeseaaesseneessas 12 
2.5 DETECT AND AVOID ALERTING LOGIC FOR UNMANNED SYSTEMS ......scceescecssscesseeseseeseseeceeeessaeesssaeesnees 13 
2.6 PILOT, MODEL: tesetecticctcae tase letei becace ds esi eta caste ete thle nce eect ra ats i dest Se 15 
2.7 UAS PERFORMANCE ASSUMPTIONS ......csscceessecesscessecessuecsssaececeeecaeecsueeessuecseseecseeeseesssaaessessesseneessaress 16 

Be. METHOD eisssstss sateccrcssas cewencs cea wscat ee sate ut ateidehehebecataan ats Sacha eacececibed Sate ctactahe sce ide decastcecateestemantecs 17 
3.1 SCENARIOS Sveei-becfes Joss P eat tat Meee ak ei isos ake at cree ieee a eae ling tena 17 
3.1.1 National Airspace System (NAS)-Derived ENCOUNTEr SetS .......ssccccesssssccesssssecssssssecessssecesnsssees 17 
3.1.2 MOPS Requirements-Derived Test VeCtOIS ......cssssccccccecsssssssssececssnsnssascesecscsesensssessseceesnssasasees 18 

3.2 IVIETRICS 25 26e Pay athe Aeec fale co deiteti tetas easel tn Ate aden deed sedis wih ete oe ote nasties 20 
3.2.1 Severity of Loss of Well Clear (SLOWC) .....:s:cccsssssccessssseceessesecsessesecessssesesssecsesssecsessseceesaseees 20 
B22” . SAICKESCONING  theccnesdovetebene ered oils obtacte eeles Pete hlos ete be Alte eben ed rated does tag ete eb eee Pe 20 
Bi2.3°' “AlCKE SILO Ri estecscessccsSencinecuse cde detest detanals dedin dane Sunk Psd ddan dened eu a hee Saaak iat cated 24 

4.. SUMMARY OF RESULTS. viv ssccscsiscscssecsdessachcssedssbsevescsescbsssentssctessnsbestssesssdccnsbosssbebeaseddusbsesssvSeadsese 25 
4.1 SEVERITY OF LOSS OF WELL CLEAR (SLOWC) ........cc:cccccsssscccssscececssscececsssseeecssesecscsssecesssecsceseseesesaaess 25 
4.2 ALERT SCORING s.3iice; ees cant oidadbiel calieeeetaat ions reiveievetisieln tile adele ie atv deahinia ach dan 29 
4.2.1 Corrective Alert SCOLIING......cssssscccccessseessnsssececssesnenssssecscssnsesassesecssuseenuassesscesessasaesesscsseeneaaa ees 30 
4.2.2 Warning Alert SCOLING ......ccssesssscccesessesssececccesessesasseecssssnessaaeseseceseseauasssessuseessassasssesseeeeauaaseens 33 

4.3 ALERT JITTER OPEN ‘LOOP Siedrrcceste stent viele ies taiteren tds era eit athe ties lout tee eget: 36 
5. NATIONAL AIRSPACE SYSTEM (NAS)-DERIVED ENCOUNTER SET RESULTG..............:ssssscssssssseeseees 39 
5.1 CORRELATED: ENCOUNTERS i: 2302 :05 295 e0045:08-c0ssbobveasdi eteiavedessiedcesdin avn ea vdid dues bec ven Tivesslbilaieaslneadiadis 39 
5.1.1 Severity of Loss of Well Clear (SLOWC))......::cccsssscccssssssecesssssecssssssecsessseseassecssassecsesssecsessssecs 39 
S12: AICrtsSCONING sedsseteccses sce ccdes sbackaedvve ape ves scdee Conv steveueads Coaavévalcahebes avandde telausetvedecsdelieeiés fuaiteeaveoebans 41 
Sili3:  -Al@rt JItOr Lesccevcastcevcssiavccatesieccdstdencdvelavadaveedcedisoteedisstavelévbalecdénsiecdéeslacnddesidcecesstesesssdavestleasévelis 45 

5.2 UNCORRELATED ENCOUNTERS.......ssesccssssecssseessssecssseesseecsssuecsececsessessesessauecseseusseusasesesaecseauessesesseneess 47 
5.2.1 Severity of Loss of Well Clear (SLOWC))......::cccsssscccessssseceessssecsessssecsssssesesssecseassecsessssecssnssecs 47 
52.2. AICrtSCONING ssi: de2cccsessdevsads senceseesda vive venycieesblavcaPaeyedds iiinedsdnstsets avescdsUiiaesatviavcstelaedise vuaitecsvcbeeues 49 
Bi2.3. -Al@rt JIC ecceccactcevscsiavecivadecsdvidevsdeelaeadivendesdasetescusetabedicbalenséesieedéeslactdasiiesesssbivesvelaventlerténsde. 53 

6. MOPS REQUIREMENT-DERIVED TEST VECTORS RESULTS. ........cccccccssssssssssesssccccessscssccsssesssesssessees 55 


6.1 SEVERITY OF LOSS OF WELL CLEAR (SLOWC) ........cc:cccccssssccesscececssscececssseececsseseescssesecesnsseesesseaeesesaaess 55 


OFT  ROGGISLOW Co eosck cobs caacte ch cecosc ava dat du cssnets stone these oadbesoethe siisenas ca cacesit duossade vacesessuaades ivccpoeneton’s 55 


6.1.2. Active Surveillance Transponder (AST) SLOWC.......::ccccssssccessssceceessccecesssececessececessasseeessseeeeees 58 
6.1.3 Automatic Dependent Surveillance — Broadcast (ADS-B) SLOWC.........sssccccessssecessssceeessseeeees 62 

6.2 ALERT. SCORING 220s <:c2sccccoeSacevuzh evadesacan eaves suateiuscendtsavestecnstueveasteus stizse  cwevielevtstaataieeten dard verte eestes 65 
6:2:1 “COrrective: AlErt SCOMING sccezcccs cages toads sos tebe s sa atts 3 capes Soa cea he ATEN Saeco Hedda waa Fea wR HE 65 
6:2:2°°  Warnitig- Alert: SCOPING ssszzlis<scechsotecchg. 5 aceiys eaccnd os eectan teacttys sass age ahaa aves ie aves Rhos oanTlee eeaIaSs eT 76 

6.3 AMERT: JITTER fe ev:cvesaresseceeoeteecsegsasies fu steasteceatariate dastet ence eeiaet Povitens odigsend eee caeati geet dace ckeseandee en teseehes 87 
G:3:1 -RAGGPALIErEIIELON, v..s:2 cise steaceeacahaslcatta eeschag ev taadinaavsates sescbldssescndad ebsaandesencadaassadbadsesidesencaneuaints 87 
6.3.2. Active Surveillance Transponder (AST) Alert Jitter .......ccsccccccsssssccesssseceessssecesssesecesssssecsessseces 90 
6.3.3 Automatic Dependent Surveillance — Broadcast (ADS-B) Alert Jitter ......:ccccccscccccssssscceesssseees 94 

6.4 “OFF-NOMINAL™:CASES ..2eacsucsszcecacesczeaceseucates ashes ehssedtanedacbacussbecevesyoueuaatareavatsheshahsveteaeavenciwes sabesans¥ers 98 
CONCLUSION 5s iiccseeseiccdecuecdeleversicistied cvanciviedsobercuesnsstecdcasteeeccstsesdddeschedetecsevcdcdenssstdveusestesdarsesecuses 107 
ACKNOWLEDGEMENTS vss sites cccevevccd dict cccdensseiettevessdcteceaveusrscedscescsctesssiecdatececvchdessscsstecsdusesteveescessduses 108 
REFERENCES bvviccceesccceve seceded cccnteccvciesducvecvetsscnvtserscedstetotccsessdeccadssdécdtevetsiecddtesscnctdersseuctesteccussecsdstssedeces 109 
APPENDIX A — DEPICTION OF ENCOUNTER GEOMETRIEG..........sccssccssccssccsccnsccnsccesccssccsscesscessesssesseees 110 
HEAD-ON ENCOUNTER GEOMETRY ....sccsssscceescecssececueceseecscseeceeeeessuecsssaeceeecsesesaeeessuecseaseceeeeesesessueesssaessenees 110 
CROSSING ENCOUNTER GEOMETRY .....sccssssscsescecesececsuesesuecseseeceeeeecsuecessaessceecseueesaeessaaeeseseeceeeesaeessaaeeseaaeeees 111 


OVERTAKE ENCOUNTER GEOMETRY: 2. 2: 2.csssatancvs-sasdeacacusenehcnasavenededenaedaaceberancsdedhdeteaeasung evs ebdedibegieleedednesepertees 112 


Table of Tables 


Table 1 Ownship-GRs Error Model. Parameters s.asccscccceiyuictaie visas cn sede sates seeatietacensteaavdavanddaeisetes 7 
Table 2. Radar Emulator Model Parameter..............::ccscccccsessessesesentsonssncsescesesenegensesceseesesenaeseceets 8 
Table 3.0) Emulator Model Paranicters in i.cccsssecosestvgceedivanacs ade cepsiu aes akgude deoagasicaspabaeronpune eeeaavanes 9 
Table4,: ADS-B: Fmulator Model Parameters ts sats iasiis atch datl teas salccseauy sunteadanniee aan 10 
Table 5. DAIDALUS Buffered Well Clear Definition... cece ceseeseeneeeeeeeceeecesecneeeaeeeeeeereaees 14 
Table 6. Unmanned Aircraft Performance ASSUMPTIONS ............cccccceceseceteceeteceeeeeeseeceaeenseeneeeenaees 16 
Pable.7; Final. F2-V2 "Test: Vectors: Sets cy vitae yur aeuret asa eae eae wee ea 19 
Table of Figures 

Figure 1. E2-V2 Architecture for Unmanned Aircraft Batch Simulation... eeeeeeeneeeeeeeees 5 
Figure 2. Notional Depiction of Sensor Uncertainty Mitigation Approach............c:ceceeeeeeseeeees 12 
Figure 3. 2-2 PrOmViGd elo ysstesret cei siee Siechne sy acheter aan sris eee ee nea eae eae 15 
Figure 4. Notional Depiction of Hazard, May Alert, and Non-Hazard Zones...........c:ceseeseeseees 21 
PIpUre D7 AISI SCOTS PLOCESS susiysssukeis waealegs obi cars in counties thu ani lepasdineoe sacs magtaiedade sasaesaoae duncan stesasstoe 22 
Figure 6. Radar SLoWC for Combined Encounter ..........cccesceesesseeeseeseceeeceneceeeeaeeeeeeeaecneeeaeeeneess De 
Figure 7. Radar SLoWC for Combined Encounters that Lost Well Cleat... .ecceceeceeeeeteeneeens 26 
Figure §-AST SLoWC for Combmed Pncounters:c.cece.eseacis sete issy secs ceeh eee dictes Weateea ese nancies 26 
Figure 9. AST SLoWC for Combined Encounters that Lost Well Clear.........ececeeceeseeseeteeneeens af 
Figure 10. ADS-B SLoWC for Combined Encounters............. ccc ceseeseeeeceseceseceeeeeeeseeeeeesecnaeeneeess 28 
Figure 11. ADS-B SLoWC for Combined Encounters that Lost Well Clear... cece eeeeeeereees 28 
Figure 12. Radar Corrective Alert Scoring for Combined Encounters............cceseeseeeeeseeeteeneeees 30 
Figure 13. AST Corrective Alert Scoring for Combined Encounters............:ccceeceeeceteeeeeneeereees 31 
Figure 14. ADS-B Corrective Alert Scoring for Combined Encounters «0.0.0.0... ceeceseeeteeneeeneees 32 
Figure 15. Radar Warning Alert Scoring for Combined EncounterS............:ceseeceeeeceteeneeeneeeneees 33 
Figure 16. AST Warning Alert Scoring for Combined EncounterS..............ccceseeseeseeeeeeeeeeseeneeees 34 
Figure 17. ADS-B Warning Alert Scoring for Combined Encounters ..0....0....cceeceeseeteeeeeneeeneees 35 
Figure 18. Open Loop Radar Alert Jitter for Combined Encounters ............ccceseeceeseceteceeeeneeeeeees 36 
Figure 19. Open Loop AST Alert Jitter for Combined Encounters «00.00.00... eseeseeseereeeeeeseeeseeneeees 37 
Figure 20. Open Loop ADS-B Alert Jitter for Combined Encounters «000.0... cceeceeeeceteeneeneeeneees 38 
Figure 21. Radar SLoWC for NAS-Derived Correlated Encounters...........ccccesssseeseceteeeeeeneeeneees 39 
Figure 22. AST SLoWC for NAS-Derived Correlated Encounters............ccsceseeseeseeeeeeeeeeeeneeens 40 
Figure 23. ADS-B SLoWC for NAS-Derived Correlated Encounters ..0........c:ceeceesceteeeeeneeeneees 40 
Figure 24. Radar Corrective Alert Scoring for NAS-Derived Correlated Encounters ................. 4] 
Figure 25. AST Corrective Alert Scoring for NAS-Derived Correlated Encounters .................. 42 
Figure 26. ADS-B Corrective Alert Scoring for NAS-Derived Correlated Encounters............... 42 
Figure 27. Radar Warning Alert Scoring for NAS-Derived Correlated Encounters ................4+ 43 
Figure 28. AST Warning Alert Scoring for NAS-Derived Correlated Encounters ..........0.....+ 44 
Figure 29. ADS-B Warning Alert Scoring for NAS-Derived Correlated Encounters.................. 44 
Figure 30. Radar Alert Jitter for NAS-Derived Correlated Encounters... ccceeeeeceseeseeneeeneees 45 
Figure 31. AST Alert Jitter for NAS-Derived Correlated Encounters........0....ccceeccesceteeeeeneeeneees 46 
Figure 32. ADS-B Alert Jitter for NAS-Derived Correlated Encounters ............eceeeeseeeteeneeeneees 46 


Figure 33. 
Figure 34. 
Figure 35. 
Figure 36. 
Figure 37. 
Figure 38. 
Figure 39. 
Figure 40. 
Figure 41. 
Figure 42. 
Figure 43. 
Figure 44. 
Figure 45. 
Figure 46. 
Figure 47. 
Figure 48. 
Figure 49. 
Figure 50. 
Figure 51. 
Figure 52. 
Figure 53. 
Figure 54. 
Figure 55. 
Figure 56. 
Figure 57. 
Figure 58. 
Figure 59. 
Vectors... 
Figure 60. 
Vectors.... 
Figure 61. 
Vectors.... 
Figure 62. 
Vectors... 


Figure 64. 
Vectors... 
Figure 65. 
Vectors... 
Figure 66. 
Vectors... 


Figure 68. 
Vectors... 


Radar SLoWC for NAS-Derived Uncorrelated Encounters............cccccececeeeeeeseeeeeeeeeeee 47 


AST SLoWC for NAS-Derived Uncorrelated Encounters...........ccceceseeseereeeeeeeeeeees 48 
ADS-B SLoWC for NAS-Derived Uncorrelated Encounters ............eseseeseereeeeeeeees 48 
Radar Corrective Alert Scoring for NAS-Derived Uncorrelated Encounters ............. 49 
AST Corrective Alert Scoring for NAS-Derived Uncorrelated Encounters ............... 50 
ADS-B Corrective Alert Scoring for NAS-Derived Uncorrelated Encounters........... 50 
Radar Warning Alert Scoring for NAS-Derived Uncorrelated Encounters ................ 51 
AST Warning Alert Scoring for NAS-Derived Uncorrelated Encounters .................. a2 
ADS-B Warning Alert Scoring for NAS-Derived Uncorrelated Encounters.............. 52 
Radar Alert Jitter for NAS-Derived Uncorrelated Encounters...........ceeceeseeeeeeseeeeees 53 
AST Alert Jitter for NAS-Derived Uncorrelated EncounterS...........cccecsseeseeeeeeeeenee 54 
ADS-B Alert Jitter for NAS-Derived Uncorrelated Encounters ...........:ceceseereeeeeeees 54 
Radar SLoWC for Head-On MOPS Requirement-Derived Test Vectors...............0+ re 
Radar SLoWC for Converging MOPS Requirement-Derived Test Vectors............... 56 
Radar SLoWC for Maneuvering MOPS Requirement-Derived Test Vectors ............ 57 
Radar SLoWC for Overtake MOPS Requirement-Derived Test Vectors...............4 57 
AST SLoWC for Head-On MOPS Requirement-Derived Test Vectors .............:006 58 
AST SLoWC for Converging MOPS Requirement-Derived Test Vectors................ 59 
AST SLoWC for High-Speed MOPS Requirement-Derived Test Vectors................ 59 
AST SLoWC for Maneuvering MOPS Requirement-Derived Test Vectors .............. 60 
AST SLoWC for Overtake MOPS Requirement-Derived Test Vectors .............::06 61 
ADS-B SLoWC for Head-On MOPS Requirement-Derived Test Vectors................. 62 
ADS-B SLoWC for Converging MOPS Requirement-Derived Test Vectors ............ 63 
ADS-B SLoWC for High Speed MOPS Requirement-Derived Test Vectors ............ 63 
ADS-B SLoWC for Maneuvering MOPS Requirement-Derived Test Vectors.......... 64 
ADS-B SLoWC for Overtake MOPS Requirement-Derived Test Vectors................. 64 
Radar Corrective Alert Scoring for Head-On MOPS Requirement-Derived Test 

patho esti panied laths scala aunt vases sca tigtel Bsaiel dtu ccchiel dala nacaseay uansh a elen a und eae a 65 
Radar Corrective Alert Scoring for Converging MOPS Requirement-Derived Test 

cise taasb tnt Se Cab Sasila cate ba he dcn Salad eCee hetaliunaalaas ld Ged ols Seated asunbicsatchscede Mita Maiucecdidtaes dis seses 66 
Radar Corrective Alert Scoring for Maneuvering MOPS Requirement-Derived Test 
naihapieg ova licoes cdeteauh ye shuleaates We Mista ness aay eran radiae a teagan gas Uaeseatuedemas biking asap Veapeuuady 66 
Radar Corrective Alert Scoring for Overtake MOPS Requirement-Derived Test 

tap aN Sa ia Pies led au Se lala 0 cna admit elon Rue balay wide ea ease aaa 67 
AST Corrective Alert Scoring for Head-On MOPS Requirement-Derived Test Vectors 
thal goitehs Sb ca Slsaidd iden da ca tar od adt ls ce esa Acaet walgsos a tnut's a Dnaissaa 0a ee Re ale Cede eaten Snctisreh earns 68 
AST Corrective Alert Scoring for Converging MOPS Requirement-Derived Test 
catdasadphabexdish sesniass poeilanediad urateusvakevustuahaaiieauama labia. stud pasted eteaw ssmeee tems uhtae talons 69 
AST Corrective Alert Scoring for High Speed MOPS Requirement-Derived Test 

pis Aaadsiayacs tN anodehas atau Ac duet maa oenediti hear shaeine Nuldem ution dials kya uinint Neale chaser aunt Ats 69 
AST Corrective Alert Scoring for Maneuvering MOPS Requirement-Derived Test 

sass taneaseasualenasiaaaeaucsedu racing etehuseaacaiseupvahtapbonbaiss dpsed ipunledautaae jute Di asruchtasbrecy Oe vassheashenatal! 70 
AST Corrective Alert Scoring for Overtake MOPS Requirement-Derived Test Vectors 
Eales wf Steet Pa ahead esta ites ted et ta paetog eae ta ease Saww Sd allay nea ae ipei teen Nastia cas cones teas 71 
ADS-B Corrective Alert Scoring for Head-On MOPS Requirement-Derived Test 

sae bia leet ied Siete ae tian N gals italeiteta ed Bes te tainty te dcaiiteteatots teed ca tiias ete Cadel ibs sales tus supe ea leeate ted itekes 72 


Figure 69. ADS-B Corrective Alert Scoring for Converging MOPS Requirement-Derived Test 


NCCI Tsrete patna teeta ned tuna eau cma neato tu mictetia asd atts baie aca acu ta lsacgeionun Mas Justia neted ep amionsn ae cc da, d3 
Figure 70. ADS-B Corrective Alert Scoring for High Speed MOPS Requirement-Derived Test 
NICCEOIS seas ey saias ge Destaod Vi sntias pases ty sua tag stescE an enaiilgh edu cbes estes ts pltug sede Seana sR hence ie Rede Sacdag lta a adetens 74 
Figure 71. ADS-B Corrective Alert Scoring for Maneuvering MOPS Requirement-Derived Test 
MGCLOES 25: Po sis Sous dai sateinpa lea recpic decay sardacgusksesw eas ab scatecuneouiiasas detesks cotdast pac teaieings suadececaunaetaadaassatiautuan’ 74 
Figure 72. ADS-B Corrective Alert Scoring for Overtake MOPS Requirement-Derived Test 

V COLORS cane, See ini tua daaet eau etdaadn ha edeieetin ete Metaias pape asa ss eas aes sapeee et 75 
Figure 73. Radar Warning Alert Scoring for Head-On MOPS Requirement-Derived Test Vectors 
Tugeaea st dsst Se tasabias ly basbaup acaprnosiiel uate cag bnenatusa sis tnshcnarssed sade tas Neus daae dndash ull eg aadesdhasavediesiaeavnctenaid Gatana ite 76 
Figure 74. Radar Warning Alert Scoring for Converging MOPS Requirement-Derived Test 

IV ECU satruds wiaactt tahon Gale ecke savanna saaaticgaieytada aust opaalin ins wae tce agantieano aunt dane sen ava geusaunmensunmecnahen senate 77 
Figure 75. Radar Warning Alert Scoring for Maneuvering MOPS Requirement-Derived Test 

MC CUES 24 5ts cota lve A it Sasi tdosk tah ca uate See datas etude aa tenes ta Sata aa Pete demas a wad gee Sule pees afta aanetes ce: 
Figure 76. Radar Warning Alert Scoring for Overtake MOPS Requirement-Derived Test Vectors 
bles eGeei dicts essed Saitiee ah sone vosmuseanl ca cqiisS aida ac stouss Siete iea Miu ese ake Samsara Mel emia teeta 78 
Figure 77. AST Warning Alert Scoring for Head-On MOPS Requirement-Derived Test Vectors 
Gl ad eee Lit nate atacand ated halen ata eg hO laa aie ate ta ie dian at tesla tala edad cee ai tue git tbe Silaag 79 
Figure 78. AST Warning Alert Scoring for Converging MOPS Requirement-Derived Test 

NEC CLOIS i sia ey Sadat ga cinnen ag peniact vo aatecaag sues vaste ae asus’ seatteo as gaucade oi datagdau once 94 bs ou ua w aan aa geaelasdataia viet tas 80 
Figure 79. AST Warning Alert Scoring for High Speed MOPS Requirement-Derived Test 
MECUONS causa ctil aus Sactes ieineits pecdaal telenSealtuaty WkGeinwes vatutan asysniae ea seaseatecma es acer elea sea tora eaSahsyoestenetiv ates 80 
Figure 80. AST Warning Alert Scoring for Maneuvering MOPS Requirement-Derived Test 

V COLON sa ads sets saiSacagrcesgsvudsanecaes seetussiege suas castes foeiscasoada eal pi veasguelikdancesaedscaeduvadeaii suas Candeawaedisesoned 81 
Figure 81. AST Warning Alert Scoring for Overtake MOPS Requirement-Derived Test Vectors 
situitcosualteststesSacoussnitbubant ielaiaxstncphaunumslaneteeish veda sslayatoathastasisicassibasbundail ssusehdapestalehasbaaidahanduaewuadh canbindusta 82 
Figure 82. ADS-B Warning Alert Scoring for Head-On MOPS Requirement-Derived Test 
COUT S Ss rinse natird iUa aS ater dtd 4d us des and grape cag ae BOG aR cht cal dea Beh ancl gis AAG aa catael Las ig 83 
Figure 83. ADS-B Warning Alert Scoring for Converging MOPS Requirement-Derived Test 

NM COWES i srs oc sista sa te apadtena avg Sins ln tase tel gates lana hetiede aus ccascleuaisi ata ech gelatin pera ics shee eiia ns Cinautstaaes oe 84 
Figure 84. ADS-B Warning Alert Scoring for High Speed MOPS Requirement-Derived Test 
VOCLOMS sessieashcaecesanssnvyiccanebig eames eycassie con ieats we pabuees erty beni nbeeiaaatesfs nstoes Wanngueanee ty eaanonneaousebes 84 
Figure 85. ADS-B Warning Alert Scoring for Maneuvering MOPS Requirement-Derived Test 

MV CCLOUS al aiea esis ncubaes ehh els ga catia edie shgnand atw'tas atecaa ee teana nie wg a aata cay deepen dase ts wipaeed nuts siudiesmand heats 85 
Figure 86. ADS-B Warning Alert Scoring for Overtake MOPS Requirement-Derived Test 

NC CTOIS 9 ihc eceio dies ae deste Sasa ess Ses Aa each shoe bau estes Oecd dae eatin pede catia Succes Esco day weladies Rescate Suan odaaites 86 
Figure 87. Radar Alert Jitter for Head-On MOPS Requirement-Derived Test Vectors............... 87 
Figure 88. Radar Alert Jitter for Converging MOPS Requirement-Derived Test Vectors........... 88 
Figure 89. Radar Alert Jitter for Maneuvering MOPS Requirement-Derived Test Vectors. ........ 88 
Figure 90. Radar Alert Jitter for Overtake MOPS Requirement-Derived Test Vectors............... 89 
Figure 91. AST Alert Jitter for Head-On MOPS Requirement-Derived Test Vectors..............4. 90 
Figure 92. AST Alert Jitter for Converging MOPS Requirement-Derived Test Vectors............. 91 
Figure 93. AST Alert Jitter for High Speed MOPS Requirement-Derived Test Vectors............. 91 
Figure 94. AST Alert Jitter for Maneuvering MOPS Requirement-Derived Test Vectors.......... 92 
Figure 95. AST Alert Jitter for Overtake MOPS Requirement-Derived Test Vectors...............+ 93 


Figure 96. ADS-B Alert Jitter for Head-On MOPS Requirement-Derived Test Vectors ............ 94 


Figure 97. ADS-B Alert Jitter for Converging MOPS Requirement-Derived Test Vectors. ........ 05 


Figure 98. ADS-B Alert Jitter for High Speed MOPS Requirement-Derived Test Vectors ........ 95 
Figure 99. ADS-B Alert Jitter for Maneuvering MOPS Requirement-Derived Test Vectors...... 96 
Figure 100. ADS-B Alert Jitter for Overtake MOPS Requirement-Derived Test Vectors .......... OF 
Figure 101, "Track History -for Fneounter 304 vgs acacia wvaageseer eunudesincatp rae oennous: 98 
Figure 102. Histogram of all runs for Encounter 304 0.0... eeceeceseesseeneeeeeeseeeeeeeeeeseceaeceeeaeeeneess 99 
Figure 103. Alerting and Guidance Time History for Encounter 304 with Truth Surveillance Data 
ssl pact os daa tthaats unr eas sutyl tas Maan tat elena doce ies fasl cts cast vi une Se Sul ca hit st steed Moyer A ahha Vince sins 100 
Figure 104. Alerting and Guidance Time History for Encounter 304 Sensed ADS-B Surveillance 
SSSI OM Sy asics chap saontth ous lise ande Neca snsuases seu decs cpamaey saunter a Sunes buts a assuaties ata cud es Buea tebaata gnats asad ots eatnisansauen 101 
Figure 105. Alerting and Guidance Time History for Encounter 304 with Mitigated ADS-B 
SUPVEIIANCES CSO ain cataris eats circa anwseod icsnsaece. astinaltana tavodg dey ain comana Avtunse teepanie culinteauselmetily 102 
Figure 106. Track History for Encounter 216 with AST Surveillance Sensor... cece 103 


Figure 107. Maximum SLoWC Histogram for Encounter 216 with AST Surveillance Sensor. 103 
Figure 108. Alerting and Guidance Time History for Encounter 216 with Truth Surveillance Data 
pestis dy eis te catia ee Dandatiaa eyes ues eA Dats sed Sia ah Ocak cies a ty ota slehr eats unre tesk na nau aus yaueeiuhs 104 
Figure 109. Alerting and Guidance Time History for Encounter 216 with Sensed AST 

SUEY CHIANG CS CUS OR oii ciel dasa eneterecte ais dan ponteats egal aap mavens taal es 8d decyl ra assess 105 
Figure 110. Alerting and Guidance Time History for Encounter 216 with Mitigated AST 

SUT VSI ANCE SMSO secs ctsede adidas ey Seguin er ahin ct omaailde sl eh sngliea st eae as ooasicee ude deniea Gace adeeb Unt senaeeie Vanes 106 


List of Acronyms 


2PAIRS 2 degrees-of-freedom Prototyping Aircraft Interaction Research Simulation 
ADS-B Automatic Dependent Surveillance — Broadcast 
AGL Above Ground Level 

AST Active Surveillance Transponder 

ATC Air Traffic Control 

CPA Closest Point of Approach 

DAA Detect and Avoid 

DAAET DAA Execution Threshold 

DAAWC DAA Well Clear 

DAIDALUS _ Detect and Avoid Alerting Logic for Unmanned Systems 
E2-V2 End-to-End Verification and Validation 

EKF Extended Kalman Filter 

FAA Federal Aviation Administration 

GA-ASI General Atomics Aeronautical Systems Inc. 

GPS Global Positioning System 

HAZ Hazard Zone 

HAZNot Non-Hazard Zone 

ICAO International Civil Aviation Organization 

LaRC Langley Research Center 

LoWC Loss of Well Clear 

MAZ May Alert Zone 

MIT/LL Massachusetts Institute of Technology’s Lincoln Laboratory 
MOPS Minimum Operational Performance Standards 
msec Millisecond 

NACp Navigation Accuracy Category Position 

NACv Navigation Accuracy Category Velocity 

NAS National Airspace System 

NASA National Aeronautics and Space Administration 
NMAC Near Mid-Air Collision 

nmi Nautical Mile 

RTCA Radio Technical Commission for Aeronautics, Inc. 
SARP Science and Research Panel 

sec Seconds 

SLoWC Severity of Loss of Well Clear 

SUM Sensor Uncertainty Mitigation 

TCOA Time to Co-Altitude 

UA Unmanned Aircraft 

UAS Unmanned Aircraft Systems 

UAT Universal Access Transceiver 


WC Well Clear 


Abstract 

As Unmanned Aircraft Systems (UAS) make their way to mainstream 
aviation operations within the National Airspace System (NAS), research 
efforts are underway to develop a safe and effective environment for their 
integration into the NAS. Detect and Avoid (DAA) systems are required to 
account for the lack of “eyes in the sky” due to having no human on-board 
the aircraft. The current NAS relies on pilot’s vigilance and judgement to 
remain Well Clear (CFR 14 91.113) of other aircraft. RTCA SC-228 has 
defined DAA Well Clear (DAAWC) to provide a quantified Well Clear 
volume to allow systems to be designed and measured against. Extended 
research efforts have been conducted to understand and quantify system 
requirements needed to support a UAS pilot’s ability to remain well clear 
of other aircraft. The efforts have included developing and testing sensor, 
algorithm, alerting, and display requirements. More recently, sensor 
uncertainty and uncertainty mitigation strategies have been evaluated. 


This paper discusses results and lessons learned from an End-to-End 
Verification and Validation (E2-V2) simulation study of a DAA system 
representative of RTCA SC-228’s proposed Phase I DAA Minimum 
Operational Performance Standards (MOPS). NASA Langley Research 
Center (LaRC) was called upon to develop a system that evaluates a 
specific set of encounters, in a variety of geometries, with end-to-end DAA 
functionality including the use of sensor and tracker models, a sensor 
uncertainty mitigation model, DAA algorithmic guidance in both vertical 
and horizontal maneuvering, and a pilot model which maneuvers the 
ownship aircraft to remain well clear from intruder aircraft, having 
received collective input from the previous modules of the system. LaRC 
developed a functioning batch simulation and added a sensor/tracker 
model from the Federal Aviation Administration (FAA) William J. Hughes 
Technical Center, an in-house developed sensor uncertainty mitigation 
strategy, and implemented a pilot model similar to one from the 
Massachusetts Institute of Technology’s Lincoln Laboratory (MIT/LL). 
The resulting simulation provides the following key parameters, among 
others, to evaluate the effectiveness of the MOPS DAA system: severity of 
loss of well clear (SLOWC), alert scoring, and number of increasing alerts 
(alert jitter). The technique, results, and lessons learned from a detailed 
examination of DAA system performance over specific test vectors and 
encounter cases during the simulation experiment will be presented in this 


paper. 


Introduction 


As the number of Unmanned Aircraft (UA) continues to increase, the demand to allow them 
routine access to the nation’s airspace drives research into methods and technologies to ensure that 
their integration into the National Airspace System (NAS) is done in a safe and efficient manner. 
One of the main challenges to safe integration is development of a capability for the unmanned 
aircraft system (UAS) to detect and avoid other air traffic. A Detect and Avoid (DAA) system 
would overcome the UAS’s lack of “eyes in the sky.” Understanding key DAA system 
performance capabilities and limitations is essential to developing rules and regulations that 
prioritize safety as UASs are integrated into routine NAS operations. 


To date, DAA research has focused on sensors to detect other aircraft, algorithms to help make 
avoidance determinations, and displays to help the operator remain well clear of other aircraft (as 
required by CFR 14 91.113). To enable a DAA capability, a mathematical definition of DAA Well 
Clear (DAAWC) has been developed that provides a minimum separation volume for large UA 
operating in the NAS. A federal working group known as a Science and Research Panel (SARP) 
on UAS that consists of representatives from the Department of Defense, Department of Homeland 
Security, the Federal Aviation Administration (FAA), and the National Aeronautics and Space 
Administration (NASA) developed the DAAWC definition at the request of RTCA Special 
Committee 228 (SC228). RTCA “functions as a Federal D Committee and develops consensus- 
based recommendations on contemporary aviation issues” (RTCA, Inc., In Press). SC-228 consists 
of government, private, industry representatives, and other interested parties who are working to 
develop the Phase I Minimum Operational Performance Standards (MOPS) for DAA equipment. 
The initial phase of standards development (Phase I) focused on civil UAS “transitioning to and 
from Class A or special use airspace” (above 500 feet Above Ground Level (AGL)), and 
“traversing Class D, E, and G airspace” in the NAS (RTCA, Inc., In Press). Phase I DAA MOPS 
are to be published in early 2017 as an RTCA DO publication. 


SC-228 members have conducted extensive research efforts to understand and quantify DAA 
system requirements needed to support a UAS pilot’s need to remain well clear. With proper 
design, these systems “could provide an additional layer of protection that maintains or even 
enhances the current exceptional level of aviation safety” (Kochenderfer, Espindle, Kuchar, 
Griffith, October 2008). The safety-critical nature of a DAA system requires a robust and rigorous 
assessment before confidence can exist to certify a system for operational use. Research efforts 
have included developing and testing sensor, algorithm, alerting, and display requirements. 
Recently, an evaluation was conducted on sensor uncertainty and a mitigation strategy has been 
developed and assessed (Jack, In Press). 


However, to understand these capabilities and limitations in a wider scope, NASA Langley 
Research Center (LaRC) evaluated the Phase I DAA MOPS requirements with end-to-end DAA 
functionality in order to verify and validate that a representative MOPS-DAA system performed 
acceptably. The End-to-End Verification and Validation (E2-V2) simulation study included the 
use of sensor and tracker models, a sensor uncertainty mitigation model, DAA algorithmic 
guidance in horizontal maneuvering, and a pilot model that commanded the ownship aircraft’s 
heading in response to the DAA algorithm’s maneuver guidance to remain well clear from other 
aircraft (i.e., an intruder). The MOPS-representative DAA system was evaluated over an extensive 
set of encounters representative of flight operations in the NAS, as well as several encounters 
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designed to invoke specific system responses. The resulting simulation provided the following key 
parameters, and many more, to evaluate the effectiveness of the MOPS DAA system: severity of 
loss of well clear (SLoWC), alert scoring, and number of increasing alerts (alert jitter). The 
technique, results, and lessons learned from a detailed examination of DAA system performance 
over specific test vectors and encounter cases during the simulation experiment will be presented 
in this paper. 


1. General Information 


1.1 Approach and Objective 


The primary focus of the End-to-End Verification and Validation (E2-V2) simulation study was 
to show, through detailed examination of specific test vectors and encounter cases, whether a 
MOPS-representative DAA system behaved acceptably. System performance with, and without, 
sensor uncertainty mitigation (see Section 2.4) was investigated so that relative system benefits 
may be quantified. E2-V2 served as an additional validation of a MOPS-representative DAA 
system in a closed-loop fast-time simulation environment along with a limited number of open 
loop trials. 


2. Simulation Environment 


An end-to-end fast-time simulation tool was developed to model simplified unmanned aircraft 
(UA) operations. All DAA system components were also modeled, including: an aircraft 
performance model referred to as 2PAIRS (2 degrees-of-freedom Prototyping Aircraft Interaction 
Research Simulation), various sensor models, a DAA tracker model, a deterministic pilot model, 
a sensor uncertainty mitigation approach, and a representative DAA algorithm, known as 
DAIDALUS (Detect and AvoID Alerting Logic for Unmanned Aircraft Systems). 


2.1 E2-V2 Architecture 


A notional depiction of the E2-V2 simulation architecture is shown in Figure 1. There are three 
paths for data flow within the simulation tool: 


The first path, labeled “Truth” in Figure 1, provides true (i.e., perfect) state data as output by the 
simulation tool 2PAIRS (see Section 2.2) to the DAIDALUS software (see Section 2.5), which 
outputs guidance bands to the pilot model. The pilot model then determines a heading change 
command with respect to the truth-based guidance bands. 


The second path, labeled “Sensed” in Figure 1, provides true state data to the Sensor/Tracker suite 
(see Section 2.3) which degrades the state data using sensor accuracy assumptions. After 
degradation, the state data is then provided to DAIDALUS which outputs guidance bands based 
on the degraded state data of the ownship and intruder. The pilot model (see Section 2.6) then 
determines a heading change command with respect to the degraded state based guidance bands. 


The third path, labeled “Mitigated” in Figure 1, provides true state data to the Sensor/Tracker suite 
which degrades the state data using sensor accuracy assumptions. After degradation, the state data 
is then provided to the Sensor Uncertainty Mitigation (SUM) software (see Section 2.4), which 
acts as a wrapper around DAIDALUS (i.e., the DAIDALUS algorithm is embedded within the 
mitigation model, which determines the positions and velocities of the aircraft). The SUM wrapper 
determines the positions and velocities of the ‘phantom’ intruder aircraft and provides the 
additional state data to DAIDALUS. DAIDALUS then outputs guidance bands based on the 
degraded state data of the ownship, the sensed intruder aircraft, and all ‘phantom’ intruder aircraft. 


The pilot model then determines a heading change command with respect to the resulting guidance 
bands. 


“Truth” 


DAIDALUS Pilot Model 


“Sensed” 


2PAIRS DAIDALUS Pilot Model 


FAA Tech Center 


SUM Mitigation 


DAIDALUS Pilot Model 


“Mitigated” 


Figure 1. E2-V2 Architecture for Unmanned Aircraft Batch Simulation 


2.2 Aircraft Flight Dynamics 


An aircraft performance model, 2PAIRS, was used to model the UA’s dynamics throughout the 
encounter and simulated lateral maneuvers and various ground speeds. 2PAIRS consists of a 
library that ingests either initial position and velocity or a time-sequence of positions for one 
ownship and one or more intruders. Ownship and intruder data is propagated in time according to 
these inputs until the Pilot Model (see Section 2.6) begins to output heading commands to resolve 
conflicts. At that time, ownship lateral motion is modeled using a simple 2 degree-of-freedom 
(bank and heading) model; the aircraft initiates a 3 degrees/sec turn with a 5 degrees/sec roll-in 
and roll-out rate. The library tool can capture and store key data about the encounter for further 
investigation and analysis, which is referenced at run time as a JAR (Java Archive) during batch 
simulation execution. The assumed maneuver rates and accelerations of this study did not stress 
the performance model. 


2.3 Sensor/Tracker Suite 


The E2-V2 simulation contained a version of the FAA’s reference sensor and tracker models 
developed by the FAA’s William J. Hughes Technical Center as reference models for the SC-228 
DAA MOPS. The models, written in the Java programming language, were modified to replace 
the inter-process communication between the position model and standalone sensor and tracker 
models with direct function call/return semantics for processing efficiency. The sensor model 
accepts aircraft truth-state data for ownship and intruder(s) and returns degraded ownship state as 
well as intruder target reports for specific sensor configurations. This study considered each sensor 
(ADS-B, AST, or RADAR) in isolation and did not consider multi-sensor scenarios. The tracker 
model accepts sensed ownship state and target reports and produces the final degraded track 
information needed for the system to use as input to the DAA algorithm. 


The sensor software model contains sensor emulators to perform internal detection, correlation, 
and error modeling for ownship and each intruder using truth data as input and generating sensor 
reports appropriate for each sensor type that include both the measured quantities and estimates of 
their uncertainty. The emulators for the three sensors (ADS-B, AST, and RADAR) are described 
on the following pages. 


The ownship state emulator uses a Global Positioning System (GPS) error model, shown in Table 
1, (Calhoun, 2016) to generate navigational information for position and velocity with correlated 
noise and barometric altitude. Correlated noise is generated based on the specified Navigation 
Accuracy Category for Position (NACp) and Navigation Accuracy Category for Velocity (NACv) 
values using a first order Gauss-Markov process and barometric altitude using a Laplacian error 
model. 


e Navigation Accuracy Category for Position (NACp): Parameter that “describes the 
accuracy region about the reported position within which the true position of the 
surveillance position reference point is assured to lie with a 95% probability at the report 
time of applicability” (RTCA DO-317B, 2014). 


e Navigation Accuracy Category for Velocity (NACv): Parameter that “describes the 
accuracy about the reported velocity vector within which the true velocity vector is assured 
to be with a 95% probability at the reported time of applicability” (RTCA DO-317B, 2014). 


Table 1. Ownship GPS Error Model Parameters 


Navigation Accuracy Category for Position (NACp) = 7 


State Absolute Error Bias Time Correlation 
(per aircraft) 1-sig 


Horizontal Position 75.6m 0 300 seconds 
Baro Altitude 0 Per Traffic = 
Situational 


Awareness with 
Alerts (TSAA) 
Model 


Navigation Accuracy Category for Velocity (NACv) = 1 


State Absolute Error Bias Time Correlation 
(per aircraft) 1-sig 


Horizontal Velocity 4m/s 0 300 seconds 
Vertical Velocity 1.707m/s (95%) 22 on 
Attitude [0.2, 0.2, 0.4] 22 = 

degrees 


The radar emulator (Table 2) (Calhoun, 2016) uses a single beam to search and track as it scans 
the field of view at +/- 110 degrees in bearing and +/- 15 degrees in elevation, using a specified 
pattern to generate raw measurements that include appropriate measurement noise. These 
measurements are correlated scan-to-scan to produce sensor reports. 


Table 2. Radar Emulator Model Parameters 


State Relative Error (1-sig) Bias 

Range 15.24m (50 feet) 15.24m (50 feet) 
Bearing 3.0m/s (10 feet/second) 2.4m/s (8 feet/second) 
Altitude 1-degree 0.5-degree 


Radar Emulator Model Parameters 


State Value 
Update Rate 1 second 
Tracking Range 5 nmi (< 100kts. Intruder) Detection Range Scale Factor 
6.5 nmi (100-300kts. Intruder) Az: [0, 30], 1.0 
8 nmi (> 130kts. Intruder) Az: [30, 60], 0.84 
Az: [60, 90], 0.46 
Az: [90, 110], 0.45 
Field of Regard +/- 15 Elevation (Stabilized with respect to velocity vector) 
+/- 110 Azimuth 
Probability of Track Pr(Track) = 1 


The Active Surveillance Transponder (AST) emulator (Table 3) (Calhoun, 2016) models either 
one of two interrogation modes: Mode S (with 25 feet altitude resolution) and Mode C (with 100- 
foot altitude resolution). It receives true ownship and intruder position from the true source and 
generates the AST measurements at a computed rate based on the intruder's estimated closing rate. 
Mode S maximum detection range is configured at 35 nmi with probability of detection at 95%, 
while Mode C detection is configured at 15 nmi with probability of detection at 90%. The field of 
regard in elevation is [-15, +20] degrees. 


Table 3. AST Emulator Model Parameters 


State 


Range 


Bearing 


Altitude 


Relative Error 

(1-sig) 

15.24m (50 feet) 38.1m (125 feet) 
[-10, 10 degrees]:9 — --- 

degrees RMS; 


maximum 27 
degrees 


Bias 


[-15, -10] or [10, 20 
degrees): 15 
degrees RMS; 
maximum 45 
degrees 


0 Per TSAA Model 


Quantization 


Quantization 25 ft. / 
100 ft. (intruder 
Aircraft) / 1 ft. 
(Ownship Aircraft) 


AST Emulator Model Parameters 


State 
Update Rate 


Detection Range 


Probability of 
Reception/Detection 


Field of Regard 


Value 
1 second 


Mode C = 15 nmi 
Mode S = 35 nmi 


Mode C = 0.90 
Mode S = 0.95 


[-15, +20 degrees) Elevation 


The Automatic Dependent Surveillance — Broadcast (ADS-B) emulator (Table 4) (Calhoun, 2016) 
models either one of two transmission modes: 1090 MHz Extended Squitter (1090ES) and 
Universal Access Transceiver (UAT). It uses a similar error model to the ownship state emulator. 


Table 4. ADS-B Emulator Model Parameters 
Navigation Accuracy Category for Position (NACp) = 7 


State Absolute Error Bias Time Correlation 
(per aircraft) 1-sig 


Horizontal Position 75.6m 0 300 seconds 
Baro Altitude 0 Per TSAA ane 
Model 


Navigation Accuracy Category for Velocity (NACv) = 1 


State Absolute Error Bias Time Correlation 
(per aircraft) 1-sig 


Horizontal Velocity 4m/s 0 300 seconds 
Vertical Velocity 1.707m/s (95%) --- --- 
ADS-B Emulator Model Parameters 
State Value 
Update Rate 1 second 
Latency Effects < 0.4 seconds 
(Uncompensated) 
Detection Range < 20 nmi 
Probability of 0.95 
Reception/Detection 
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The tracker software model contains emulators to perform radar, ADS-B, and AST correlation, 
sensor selection, and track smoothing. It produces filtered, geo-referenced track data and track 
uncertainty. 


e The Radar Tracker uses a three-dimensional filter to estimate intruder track states in the 
ownship reference frame. 


e The ADS-B Tracker includes position and velocity, barometric altitude, and the 
International Civil Aviation Organization (ICAO) address with time of applicability. It is 
the most stable and most accurate sensor of the three. 


e The AST Tracker uses an Extended Kalman Filter (EKF) and a polar-beta tracker to 
estimate intruder track states from AST sensor measurements. 


e Linking multiple sensor tracks into a single central/fused track for the same intruder is 
performed based on the nearest neighbor paradigm by the use of the following parameters: 
ICAO address, time-aligned sensor track positions and time-aligned sensor measurements. 
The central tracker performs a preferred sensor selection based on the propagated 
uncertainty (circular error bound computed from the elliptical track propagated covariance) 
at the time of linking. 


e The Track Manager within the Tracker outputs a track message at each one second epoch 


for each intruder and includes the linearly extrapolated central/fused track state from the 
preferred sensor track. The estimated ownship state is also output. 
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2.4 Sensor Uncertainty Mitigation Approach 


Sensor errors impose a need for a DAA system to take account of the imprecision of sensed 
intruder states when providing alerts and maneuver guidance to the UAS operator. An approach to 
account for sensor errors has been developed; the methodology is referred to as the Sensor 
Uncertainty Mitigation (SUM) model approach. The SUM model approach uses the horizontal and 
vertical position, and velocity standard deviations provided by the tracker to augment the sensed 
position of each intruder with additional ‘phantom! intruders. This approach presents the DAA 
algorithm with a block of intruders that span a sigma-multiple of the possible intruder location and 
velocity arrayed around the sensed position and velocity, as depicted in Figure 2. 
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Sensed position 
and velocity 
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Figure 2. Notional Depiction of Sensor Uncertainty Mitigation Approach 


The SUM wrapper determines the positions and velocities of the ‘phantom’ intruder aircraft. This 
is achieved by using constant scaling factors to multiply the estimated horizontal and vertical 
position and velocity standard deviations provided by the tracker model. Scaling factors with the 
SUM approach were optimized to reduce frequency and severity of losses of well clear and to 
increase probability of correct alerts: 

e Horizontal Position Uncertainty = 0.50 

e Horizontal Velocity Uncertainty = 1.50 

e Vertical Position Uncertainty = 1.00 

e Vertical Velocity Uncertainty = 1.00 
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2.5 Detect and Avoid Alerting Logic for Unmanned Systems 


The E2-2V study relied on the DAIDALUS! software library to compute maneuver guidance that 
satisfies RTCA SC-228 DAA requirements (Munoz et al, 2015). DAIDALUS functional 
specification is described in Appendix G of the Phase I DAA MOPS. The top-level functionality 
of DAIDALUS provides traffic awareness and maneuver guidance supporting UAS operators’ 
ability to detect and avoid other aircraft having the potential to cause Loss of Well Clear (LoWC). 
This functionality is intended to aid the pilot in command (PIC) in order to perform a safe 
maneuver to maintain DAA Well Clear, or regain DAA Well Clear, if a LoWC has already 
occurred (or is unavoidable). Traffic awareness is achieved through increasing alert levels, which 
correspond to increasing potential for LoWC. Maneuver guidance is provided in the form of ranges 
of maneuvers, which the pilot in command may execute to avoid or recover from LoWC. 


DAIDALUS algorithms are memoryless and were designed using truth input/output data. Further, 
the algorithms do not track or filter the aircraft input state information and do not apply time delays 
or hysteresis to the output data. Therefore, all pre-processing of surveillance input data and all 
post-processing of computed date interface is performed by external modules such as the E2-V2 
infrastructure. Under these assumptions, DAIDALUS algorithms have been formally verified for 
logical correctness (i.e., algorithms satisfy their functional requirements) and the software 
implementation has been validated against the formal algorithmic models, i.e., given a set of 
stressing scenarios, the outputs computed by the software implementation coincide to the expected 
algorithmic outputs modulo numerical imprecisions due to computer arithmetic (Munoz et al, 
2016). 


Because of the scope of the study presented in this paper, E2-V2 only exercises the maneuver 
guidance logic, the alerting logic, and the calculation of DAA Execution Threshold (DAAET) 
provided by DAIDALUS. The maneuver guidance logic in E2-V2 is configured for the class of 
aircraft that are able to perform a turn rate of 3.0 degrees/sec. In this configuration, maneuver 
guidance is computed when the current ownship state is predicted to violate, within the next 60s, 
a buffered DAAWC as defined in Table 5. Traffic aircraft are assumed to fly straight line 
trajectories. Furthermore, maneuver guidance is computed for a look-ahead time of 75s. 


The recovery guidance logic is exercised when, assuming a turn rate of 3.0 degrees/min, the 
ownship cannot avoid a violation of the buffered DAAWC with traffic aircraft. In this case, 
guidance is computed that exits the buffered DAAWC while maintaining a minimum separation 
distance of at least D-Threshold (DMOD*) horizontally and Z-Threshold (ZTHR*) vertically with 
traffic aircraft. If maintaining this separation is not possible, for example because of aircraft 
performance limits, the separation is dynamically reduced by 20% until the Near Mid-Air Collision 
(NMAC) volume. If the NMAC volume cannot be avoided, the recovery guidance logic does not 
return any guidance. 


The DAAET value is calculated when the linear projection of the ownship current state is predicted 
to violate the buffered DAAWC. In this case, DAAET is the predicted time to recovery guidance. 
In E2-V2, the DAAET is used as the time that the ownship has to avoid a LoWC assuming its 
performance limits. 
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Table 5. DAIDALUS Buffered Well Clear Definition 


DAIDALUS Buffered Preventive Corrective Warning 
Configuration Well Clear 
Volume 

D-Threshold (nmi) 1.0 1.0 1.0 1.0 
Z-Threshold (feet) 450 750 450 450 
T-Threshold 35 35 35 35 
(seconds) 
TCOA (seconds) 20 20 20 20 
Look-Ahead Time 

75 --- --- --- 
(seconds) 
Alerting Time 60 60 60 30 
(seconds) 
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2.6 Pilot Model 


The pilot model was derived from the Massachusetts Institute of Technology’s Lincoln 
Laboratory’s (MIT/LL) UAS Pilot Model Release 3.0. The pilot model was originally written in 
MATLAB/Simulink but converted to Java for this study. The Java version implements only the 
“Deterministic” mode of the model and excludes the “Stochastic” mode. The pilot model is 
implemented as a simple state machine whose states, transitions, and transition-processing 
elements are depicted in the following figure. The state transitions are triggered by timers and/or 
alert level thresholds. 


Waiting for Encounter 


Initial Delay 
ATC Coordination 


Update Delay 


Figure 3. E2-V2 Pilot Model 


The pilot model incorporates the following states: 


e The pilot model initializes in the Waiting for Encounter state. It remains in that state until 
the first update cycle in which the alert level reaches or exceeds 2 (Preventive). 


e The Initial Delay state implements a fixed (5 second) delay starting when the alert level 
first reaches 2 (Preventive) or higher, implemented using a simple cycle counter. 


e The Air Traffic Control (ATC) Coordination state is a constant (11 second) delay starting 
when the Initial Delay is satisfied, implemented using a simple cycle counter. An 
alternative transition out of the ATC Coordination state occurs if the alert level reaches 4 
(Warning) or higher. 


e The Update Delay state is a delay whose exit transition criteria is a function of the current 
alert level. It starts when the ATC Coordination Delay is satisfied and restarts each time it 
is satisfied. Maneuver Selection is performed each time the Update Delay state is re- 
entered. 


ut 


Maneuver Selection function assesses the band data (heading) and current heading to 
determine the maneuver command. It executes on each entry into the Update Delay state 
and completes its processing within the cycle that it is activated. Maneuver Selection first 
determines the minimal left and right heading deviation required to achieve the minimum 
band value (None, Recovery, Near) available in that direction. Heading changes are limited 
to 135 degrees in the MIT/LL model, but this limitation is not included in the LaRC 
implementation. Once this heading pair is determined, the logic proceeds to pick between 
the two options by first picking based on lower band value, then based on lower deviation 
magnitude, and finally picking left. Once selected, the maneuver value and selection time 
is retained until the process is re-evaluated. The vertical maneuver selection logic in the 
MIT/LL model was not implemented for this study. The LaRC implementation includes 
provision for specification of a heading band buffer that specifies how far beyond the band- 
edge the commanded heading should be selected. When the buffer is applied, logic ensures 
that if there is another band beyond the band edge, the buffer will not push the commanded 
heading beyond the midpoint of the band gap. For this study, a buffer of 5 degrees was 
used. 


The Execution Delay function implements a constant 3-second delay starting when the 
maneuver is selected. Once the execution delay is satisfied, the maneuver selection is 
passed out of the pilot model. Note that this delay is intended to represent the time it takes 
the pilot to input a command in the ground control station once a maneuver decision has 
been made, and is not related to any C2 link delay. 


UAS Performance Assumptions 


Table 6 provides the performance assumptions that were used in the E2-V2 simulation to represent 
what the UA initial state (speed) and maneuverability, in response to pilot commands, would be 
within the NAS. The model allowed the UA to instantaneously (one simulation time step) achieve 
or return to zero from the roll rate and maneuvering load factor. 


Table 6. Unmanned Aircraft Performance Assumptions 


UA Performance Capability Assumption 

Turn Rate 3.0 degrees/second 

Airspeed 40 — 600 KTAS at appropriate altitudes 
Maneuvering Load Factor Increment 0.25g 

Roll Rate 5 degrees/second 

Command and Non-Payload Control Link 400 msec delay — each way 
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3. Method 


3.1 Scenarios 


Multiple encounters were utilized to show whether a MOPS-representative DAA system behaved 
acceptability. Encounters were run in a closed loop simulation environment in order to mimic 
maneuvering behaviors of a human pilot in addition to human response delay times. A fixed pilot 
delay time, relative to alert issuance times, was used to make sensor uncertainty the only variable 
between runs of the same sensor/encounter set. Open loop encounters were also run, which 
provided the ability to characterize the original encounter geometry, with no pilot response, along 
with timing and alert jitter issues. Open loop encounters were compared to the closed loop data. 
Each closed loop encounter was run one time through the Truth mode and fifty times through the 
Sensed and Mitigated modes (see Figure 1) for replication purposes to verify and validate the 
output data. Each open loop encounter was run one time through the Truth mode and ten times 
through the Sensed and Mitigated modes. 


Encounters originated from two main categories: NAS-Derived Encounter Sets (Section 3.1.1) and 
MOPS Requirements-Derived Test Vectors (Section 3.1.2). 


3.1.1 National Airspace System (NAS)-Derived Encounter Sets 


The National Airspace System (NAS)-derived encounter sets were developed and provided by 
MIT/LL. A total of 180 encounters were provided for use in E2-V2. The 180 encounters are a 
subset of two fundamental types of encounters: Correlated (120 encounters) and Uncorrelated (60 
encounters). 


In correlated encounters, both aircraft (ownship and intruder) are equipped with a transponder and 
are, therefore, cooperative. Additionally, at least one of the two aircraft is in contact with ATC and 
will likely receive some notification about the impending traffic conflict and take action prior to 
the involvement of the DAA system. These scenarios are termed “correlated” because “the 
trajectories of each aircraft may involve maneuvers that are correlated to some degree due to this 
prior intervention” (Kochenderfer, Espindle, Kuchar, Griffith, October 2008). 


In uncorrelated encounters, at least one aircraft is not using a transponder and is, therefore, non- 
cooperative, or it involves two aircraft flying under Visual Flight Rules (VFR). Under these 
circumstances, it is unlikely that ATC would become involved prior to the close encounter. As a 
result, the two aircraft must rely solely on either visual acquisition or a DAA system to ensure safe 
separation. The assumption for this type of encounter is that “the two aircraft blunder into close 
proximity” (Kochenderfer, Espindle, Kuchar, Griffith, November 2008). 


An analysis of both correlated and uncorrelated encounters was necessary for a complete 
evaluation of a MOPS-representative DAA system. 
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3.1.2 MOPS Requirements-Derived Test Vectors 


The MOPS requirements-derived test vectors will be included as supplement to the Phase I DAA 
MOPS. Each test vector, or track, was placed in one of two categories: alerted or non-alerted. For 
the E2-V2 simulation study, only alerted tracks were utilized for data collection. “Alerted tracks 
test the alerting capabilities of a DAA system for a range of aircraft encounters that have either 
occurred historically in the en-route environment, or have been identified through flight test or the 
design of prototype DAA systems to stress the performance of a DAA system” (RTCA, Inc., In 
Press). 


The tracks were derived from multiple sources, including: 
e A review of mid-air collisions that occurred between January 2000 and June 2010 
e 95 Stressing Cases used by the Science and Research Panel (SARP) for the derivation of 
the DAAWC boundaries 
e Flight Test 4 conducted by NASA in support of DAA MOPS development 
e Test Vectors used in RTCA DO-317B for testing of the Airborne Surveillance and 
Separation Assurance Processing tracker and TSAA 


Test vectors describe cases that are representative of encounters observed during routine 
operations and categorized as Head-On, Converging, Overtake, and Maneuvering encounters. 
Additionally, test vectors also described encounters that are considered to be “corner cases” that 
stressed the performance of the system, such as High Speed encounters (RTCA, Inc., In Press). 


Table 7 shows the final closed and open loop encounter set used for each category; the final 
numbers are based on Truth tracks. Closed and open loop runs had identical encounter sets in each 
category for analysis comparison purposes. The column titled “Total” shows the initial number of 
test vectors developed. The remaining three columns show the number of encounters according to 
category description and sensor type after the initial encounters were filtered. 


Several test vectors were designed such that a loss of well clear would be unavoidable. These test 
vectors were removed from aggregate statistics. Additionally, through open loop data analysis, 
additional test vectors were removed from aggregate statistics in the following cases: 1) non- 
cooperative intruder violated operational assumptions associated with a DAA radar (RTCA, Inc., 
In Press), 2) intruder was outside of the radar or AST sensor field-of-view, and/or 3) the 
encounter’s maneuver guidance was inconsistent with lateral maneuver constraint. Appendix A 
contains notional depictions of three of the five types of encounters derived from the MOPS 
requirements. 
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Table 7. Final E2-V2 Test Vectors Set 
E2-V2 Test Vectors 


(based on Truth Tracks) 

Category Description Total Radar AST ADS-B 
NAS-Derived: Correlated 120 72 100 119 
NAS-Derived: Uncorrelated 60 44 53 56 
MOPS: Head-On 15 3 14 15 
MOPS: Converging 20 10 13 20 
MOPS: High Speed 4 0 4 4 
MOPS: Maneuvering 15 2 14 15 
MOPS: Overtake 16 3 i 16 


Total 250 134 213 245 
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3.2 Metrics 


3.2.1 Severity of Loss of Well Clear (SLOWC) 


Several metrics were used to analyze the large data set, one of which is the Severity of Loss of 
Well Clear (SLoWC). SLoWC is a metric used “to assess the severity of Loss of DAA Well Clear 
on a per-encounter basis by capturing the most serious instance of Loss of Well Clear throughout 
an encounter” (RTCA, Inc., In Press). It is based on the severity of violation into all three of the 
Well Clear components, which include: Horizontal Proximity, Horizontal Miss Distance 
Projection, and Vertical Separation. The resulting SLoWC ranges from 0% (DAA Well Clear 
maintained throughout the encounter) to 100% (mid-air collision). 


3.2.2 Alert Scoring 


Another metric used to analyze the large data set was Alert Scoring. In order for the ownship to 
remain well clear from an intruder aircraft, pilot cues, such as alerts, are needed to maintain safe 
separation. According to the Phase I DAA MOPS, “an alert is required for any encounter where 
the intruder aircraft violates the Hazard Zone at any given point throughout the encounter.” 


Figure 4 shows a notional depiction of what constitutes a Hazard Zone (HAZ), in addition to a 
May-Alert Zone (MAZ), and Non-Hazard Zone (HAZNot) and Table 8 shows the parameters for 
calculating the size of the HAZ and HAZNot zones. These hazard and non-hazard zones are used 
to define the trade space for when alerts must and must not be generated, but are not meant to 
imply a specific alerting algorithm. The hazard/non-hazard zone alert requirement structure is used 
to simplify compliance determinations without extensive analyses of alerting system performance. 
Figure 5 quantifies an alerting system’s performance. The Hazard Zone is based on the DAA Well 
Clear volume (Table 5), and an alert is required if the intruder enters this region. The May-Alert 
Zone defines a volume around the ownship aircraft in which an alert may be signaled if an intruder 
aircraft is within that volume but is not required. Lastly, an intruder aircraft within the Non-Hazard 
Zone constitutes as remaining Well Clear; no alerts should be signaled for this zone (see RTCA, 
Inc., In Press for a full explanation of alerting requirements). 


Using the HAZ and HAZNot definitions, the flow diagram in Figure 5 shows the methodology of 
scoring the alerting performance throughout an encounter. Upon completion of the simulation run, 
each alert was analyzed by comparing the time of HAZ entry and HAZNot departure are compared 
to the timing of each alert type, which is a function of sensor-degraded data. In Figure 5, the green 
region indicates where truth data is used, while the purple represents degraded alert timing 
information. 
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Hazard Zone (HAZ) 


“May Alert Zone 


Non-Hazard (HAZNot) Zone 


Figure 4. Notional Depiction of Hazard, May Alert, and Non-Hazard Zones 


Table 8. Parameters for Calculating the Size of Hazard and Non-Hazard Zones 


Preventive Corrective | Warning 
Alert Alert Alert 
Tau* mod 35 seconds 35 seconds 35 seconds 
Hazard Zone (HAZ) DMOD and HMD* 0.66 nmi 0.66 nmi 0.66 nmi 
h* (fixed) 700 feet 450 feet 450 feet 
Tau* mod 110 seconds 110seconds 90 seconds 
Non-Hazard Zone (HAZNot) DMOD and HMD* 1.5 nmi 1.5 nmi 1.2 nmi 
VMOD 800 feet 450 feet 450 feet 
Minimum Average Seconds before 
Time of Alert HWA? Wielation 55 seconds 55 seconds 25 seconds 
Seconds before 
Late Threshold (THR ate) HAZ Violation 20 seconds 20 seconds 15 seconds 
Early Threshold (THRearly) Deconds DEOK 75 seconds 75 seconds 75 seconds 


HAZ Violation 
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Using a Specific Alerting Algorithm at a Given Level of Surveillance Error 
Time Alert Was 
Issued (tajort) 
Yes Yes | Correct Required 
Alert 


FS 


Yes 
Permissible Alert 
No 
Permissible Early Permissible 
Non-Alert Alert 
Yes Legend: 
Incorrect Alert HAZ = Hazard Zone 
MAZ = May Alert Zone 


HAZNot = Non-Hazard Zone 

tuaz= Time when HAZ is initially violated 
tuaz= Time when MAZ is initially violated 
taenr= Time when Alert is initially issued 
t =0 at beginning of encounter 


Figure 5. Alert Scoring Process 


Encounter tracks with a HAZ violation are required to have an alert signaled. Together, the 
following performance metrics quantify an alerting system’s performance for required alerts: 
e Missed Alert 
Late Alert 
Short Alert 
Early Required Alert 
Correct Required Alert 


Encounter tracks within the MAZ are allowed to alert but are not required. The following 
performance metrics quantify an alerting system’s performance for MAZ alerts: 

e Permissible Non-Alert 

e Early Permissible Alert 

e Permissible Alert 
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For encounter tracks within the HAZNot, no alerts should be signaled. The following performance 
metrics quantify an alerting system’s performance for HAZNot alerts: 


Correct Non-Alert 
Incorrect Alert 


3.2.2.1 Alert Scoring Types 


Encounter tracks were scored against ten alert types (RTCA, Inc., In Press), which included: 


Correct Required Alert: Occurs for encounters “where an intruder aircraft enters into the 
Hazard Zone, and the alerting system issues a timely alert” (i.e., occurs when an alert is 
neither late, short, nor early). 


Correct Non-Alert: Occurs for encounters “where an intruder aircraft never leaves the Non- 
Hazard Zone, and the system does not issue an alert.” This type of alert is desirable. 


Permissible Alert: An “alert issued for an encounter where an aircraft enters into the May- 
Alert Zone but not the Hazard Zone, and is not early.” This type of alert is neither desirable 
nor undesirable. 


Permissible Non-Alert: Occurs for encounters where “an alerting system does not issue an 
alert and the intruder aircraft entered into the May-Alert Zone but not the Hazard Zone.” 
This type of alert is neither desirable nor undesirable. 


Early Required Alert: Occurs when “an intruder aircraft enters into the Hazard Zone, but 
the system issues an alert prior to the Early Threshold” as defined in Table 9. This type of 
alert is undesirable and is approximately the same as the boundary between the May-Alert 
Zone and the Non-Hazard Zone for non-accelerating cases. 


Early Permissible Alert: Occurs for encounters where “an intruder aircraft enters into the 
May-Alert Zone, but the system issues an alert while the aircraft still meets the criteria for 
the Non-Hazard Zone.” The logic for this alert type is based on the boundary between the 
May Alert Zone and the Non-Hazard Zone. 


Short Alert: Occurs for encounters where “an alerting system is not in alert state at the time 
at which the intruder aircraft violates the Hazard Zone, but had previously issued an alert.” 
This is an undesirable type of alert. 


Late Alert: Occurs when “an intruder aircraft enters the Hazard Zone, but the alerting 
system issues an alert less than the required time before Hazard Zone violation” as defined 
in Table 9. If an alert “is issued late and does not persist until the Hazard Zone is violated, 
(then) the alert is scored as a late alert, as lateness is considered to affect safety more 
drastically as compared to shortness.” 


Incorrect Alert: An “alert issued on an encounter for which the intruder aircraft remains in 
the Non-Hazard Zone.” For purposes related to analyzing this type of alert, incorrect alerts 
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are not referred to as false or nuisance alerts. It should be noted that, as defined, an incorrect 
alert is essentially an early alert in that it alerts the operator to an impending hazard before 
it is permitted to. 


e Missed Alert: Occurs when “an intruder aircraft enters the Hazard Zone, but the alerting 
system does not issue an alert.” This type of alert is undesirable. 


3.2.3 Alert Jitter 


Surveillance sensors (onboard the aircraft or on the ground) have inherent errors, bias and noise, 
which will skew the reported position and velocity of all aircraft in the range of coverage. A DAA 
guidance system with sensors, tracker, algorithm, filtering mechanism and displays, must be able 
to show the representative stages of alerting symbology to the pilot. These alerts must be accurately 
depicted within a timely manner, with faithful representation of airplane motions, and without 
significant jerkiness or latency (i.e., display lag, slow update rate), which would adversely affect 
the pilot’s ability to manually control the aircraft. As a result, the final metric used for analysis 
was Alert Jitter, which refers to “the average number of increasing alerting transitions that occur 
within an encounter set, where an increasing alert transition is considered to be a transition between 
no alert to any other alert level (preventive, corrective, or warning), as well as from a lower alert 
level (i.e. preventive) to a more severe alert level (i.e. corrective)” (RTCA, Inc., In Press). 
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4. Summary of Results 


To concisely present the data, the following section provides figures and discussion considering 
all encounter sets together. Focused analysis on specific encounter sets is provided in Sections 5 
and 6. The summarizing figures in this section show normalized distributions of SLoWC, Alert 
Scoring, and Alert Jitter for all encounter sets. In all figures, open loop runs are shown in black to 
show the response with no input from the pilot model. Truth runs in which the pilot model follows 
guidance based on perfect state data are shown in Blue. Runs with DAA guidance based on sensor- 
degraded state data with (Mitigated) and without (Sensed) the SUM mitigation approach in place 
are shown in Green and Red, respectively. 


4.1 Severity of Loss of Well Clear (SLoWC) 


SLoWC is a metric that quantifies the severity of the loss of separation between two aircraft. 
Figures 6, 8, and 10 show normalized distributions of SLoWC for radar, AST, and ADS-B, 
respectively. Each SLOWC distribution shown in Figure 6, 8, and 10 are with active pilot model 
response to DAA Guidance resulting in many of the encounters remaining well-clear. To provide 
further insight in the distribution of LoWC, Figures 7, 9, and 11 show normalized distributions of 
runs where LoWC occur for each sensor. The percentage of runs shown in the distribution of the 
LoWC-only figures is a function of all runs, therefore the distributions in these figures do not 
amount to 100%. 


Results for normalized SLoWC distribution for radar equipped runs (Figures 6 and 7) show that 
nearly 70% of all mitigated runs with the SUM approach incorporated results inno LoWC. Relying 
solely on sensed data without compensation for uncertainty reduces the percentage of runs that 
maintain DAAWC to less than 40%. Guidance based on truth data results in more than 80% of 
runs maintaining DAAWC, which indicates that the encounter set contains scenarios in which the 
DAA system could not maintain DAA Well Clear given ideal conditions. 
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Figure 7. Radar SLoWC for Combined Encounters that Lost Well Clear 


Figures 8 and 9 show the normalized distribution of SLoWC with AST equipped. In Figure 8, it is 
clear that the SUM mitigation approach improves the achieved SLoWC when compared to sensor 
degraded tracks only. Due to the large variations in the bearing measurement of the AST sensor 
model, the commanded heading can alternate across ownship heading. These variations in bearing 
can result in the pilot model beginning an initial turn to the left followed by a turn to the right. The 
resulting trajectory fails to achieve the required deviation from initial trajectory, which results in 
a large SLoWC values. This changing ownship course, as well as blundering intruders, accounts 
for a large proportion of the encounters with high SLoWC values in Figures 8 and 9. 
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Figure 9. AST SLOWC for Combined Encounters that Lost Well Clear 


Figures 10 and 11 show the achieved SLoWC distribution for ADS-B equipped runs. Figure 10 
shows the SUM approach resulted in fewer runs with LoWC occurrences than in Truth-based runs. 
This implies that the SUM approach adds an additional buffer within the avoidance region which 
resulted in a larger achieved separation between the ownship and intruder. Even with the decrease 
in the number of LoWC occurrences, runs with SLoWC values greater than 80% still occurred. 
Although these runs represent a small proportion of total runs (<1% of the runs using Sensed and 
Mitigated guidance), they are realistic occurrences and cannot be ignored. Beyond featuring 
blundering intruders or ownship maneuvers, these encounters are characterized by combined 
sensor-degraded state information for the ownship and intruder. The degradation of each aircraft’s 
state information can result in the DAA system sensing safe separation when in fact there is a 
LoWC (e.g., in a pair-wise encounter with the two aircraft head-on at the same altitude). The 
sensed ownship position is 200 feet above the actual ownship position while the intruder’s sensed 
position is 250 feet below the actual intruder position. The sensed vertical separation is roughly 
450 feet which may result in no alert being issued and a direct collision between the aircraft. 
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Figure 10. ADS-B SLoWC for Combined Encounters 
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Figure 11. ADS-B SLoWC for Combined Encounters that Lost Well Clear 
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4.2 Alert Scoring 


This subsection presents results showing the distribution of the categorical alert scoring outlined 
in Section 3.2.2. While the categorical alert scoring does not represent a continuum from poor to 
good performance, Figures 12 through 17 notionally order the categories from desirable 
performance to undesirable performance. The first six categories are considered desirable: Correct 
Required Alert through Early Permissible Alert. The last four are undesirable: Short, Late, 
Incorrect, and Missed Alerts. Open loop runs — runs that did not include the pilot model (i.e., did 
not “close the loop”) and therefore had no avoidance maneuvers — are shown in black for 
comparison to the closed-loop runs. Closed loop runs with Truth-based guidance and alerting are 
shown in blue, runs with sensed only alerting guidance are shown in red, and runs with the SUM 
approach are shown in green. 


Since the pilot model is responding to the DAA guidance to avoid a LoWC, the closed loop nature 
of these runs introduces an interdependency between the alert scoring of the Corrective Alert level 
and the Warning Alert level. For instance, in the following figures there is an increase in incorrect 
alerts when utilizing the SUM approach to compensate for sensor uncertainty. While this is 
undesirable behavior, there is no loss of well clear. The “Incorrect Alert” results when an alert is 
issued while still in the HAZNot zone, but the pilot model responds to this early non-permissible 
alert and commands the UA to maneuver away from the intruder. Based solely on alert scoring, 
this encounter would be viewed negatively; however, safe separation was achieved. Further, the 
Corrective and Warning HAZ zones are identical. If a blundering intruder enters the HAZ zone 
with insufficient time to allow for the DAA system to progress through the hierarchy of alert levels, 
then a Warning Alert will be issued. In the same example, the Corrective Alert will be scored as a 
“Missed Alert” due to the intruder’s blunder. Although having a “Missed Alert” is an undesirable 
behavior, the alerting system does correctly prioritize the Warning Alert over the Corrective Alert. 
In all figures below, given Truth-based guidance the pilot model responds to the alerts and is able 
to avoid entering the Corrective HAZ zone. Therefore, the provided alert is considered a 
“Permissible Alert” rather than a “Required Alert.” 
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4.2.1 Corrective Alert Scoring 


Figures 12 through 14 show the normalized distribution of the categorical alert scoring for 
Corrective Alerts for radar-only, AST-only, and ADS-only equipped runs. 


Analyzing the Truth-based open loop runs equipped with radar only, shown in black on Figure 12, 
reveals that the majority (©85%) of the runs had a “Correct Required Alert.” Of note, there were 
approximately 10% of runs that had a “Missed Alert.” Comparing Sensed runs to Mitigated runs 
with radar-quality source data, the SUM approach increased the number of permissible alerts and 
permissible non-alerts, while decreasing the short, late, and missed alerts. Undesirably, the SUM 
approach also increased the early alerts and incorrect alerts. The increase in incorrect alerts can be 
attributed to runs in which an alert was issued, but the intruder remained in the HAZNot zone due 
to ownship maneuvering. The slight increase in early and incorrect alerts was expected as the SUM 
approach introduced ‘phantom’ intruder aircraft that were perceived to be closer than the sensed 
position of the intruder. 
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Figure 12. Radar Corrective Alert Scoring for Combined Encounters 
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Figure 13 shows the normalized distribution of alert scoring for AST-equipped runs. Similar to the 
radar-only alert scoring response, the SUM approach improved the response of short, late, and 
missed alerts. Additionally, the SUM approach greatly increased percentage of runs within the 
“Early Permissible Alert” and “Incorrect Alert.” The early permissible alerts can be attributed to 
the expansion of the avoidance volume caused by the introduction of the ‘phantom’ intruders. The 
incorrect alerts were caused by the unmanned aircraft’s ability to achieve separation following the 
DAA guidance. The SUM approach also shifted the number of “Correct Required Alert” to 
“Permissible Alert” due to the increase in the avoidance region beyond the DAAWC volume. 
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Figure 13. AST Corrective Alert Scoring for Combined Encounters 


Figure 14 shows the normalized alert scoring distribution for ADS-B only runs. Similar to the 
other two sensors, the SUM approach increased the number of early permissible alerts and 
incorrect alerts. Interestingly, the SUM approach appeared to have no effect on the “Late Alert,” 
the “Permissible Non-Alert,” and the “Correct Non-Alert” responses. 


31 


Alert Scoring - ADSB - Corrective 


80 - MB Open Loop 
ee GE Truth 
= ME Sensea 
o 60 - WWE Mitigatea | 
ra) 
® 
fo) 

S$ 40; 
< 
(<b) 
4 
® 
oO 20-7 

(4) EEE =n L \ les, Te _i 

ee .@ & ee se ee ee ee & ee 
Fe oT ok oY cv ae ov ae” AY 
é < s@ x e oe & x@ 3 2 
SE SP KF EF SMS OP YK SF 

SF SF FX oo es Ss 
S oS of SS ok - 

é ee er 
x Q ie 
Alert Scoring 


Figure 14. ADS-B Corrective Alert Scoring for Combined Encounters 
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4.2.2 Warning Alert Scoring 


The following subsection contains figures with the normalized distribution of alert scoring for 
Warning Alerts for each single sensor configuration. In the majority of encounters a Corrective 
Alert was issued prior to the Warning Alert, Warning Alert being the highest level. Given sufficient 
time to respond, the pilot model will maneuver the ownship and avoid entering the HAZ zone of 
the Warning Alert. This behavior is presented in the following figures as the high percentage of 
correct non-alerts and permissible non-alerts, particularly the runs with Truth-based guidance and 
alerting. 


Figure 15 shows the normalized distribution of alert scoring for radar-only runs for the Warning 
Alert level. Comparing Truth and Sensed distributions in Figure 15, sensed tracks resulted in 30% 
more missed alerts. Using the SUM approach, the missed alerts were reduced to below 20% of the 
runs. The SUM approach effectively increased the size of the avoidance region, which resulted in 
an increase in the number of incorrect alerts (i.e., the alerts provided to the pilot model even though 
the intruder remains in the HAZNot zone). The number of late alerts and short alerts were also 
reduced, while correct, permissible, and early permissible alerts increased as a result of the SUM 
approach. 


Alert Scoring - RADAR - Warning 


100 
HB Open Loop 

sor GE Truth ; | 
2 GE sensed 
& ME Mitigatea | 
S 60- 7 
o 
fo) 
£ 
= 40 - 7 
£ 
& 

20; |. a 

Oo | | Le J F ult i 
se se se se se se se se we Se 
> QT" 6@ Rs O° 5e@ LB" @ & > 
@ ° Ne) o @ n>) ©) ys @ & 
SS = ~ = SS > ~S Vv << LP 
SS FP eS SS 2 s 
PP SF XS LY s 
ger I a ee 
o ese 
Alert Scoring 


Figure 15. Radar Warning Alert Scoring for Combined Encounters 
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The Warning Alert scoring distribution is shown for AST equipped runs in Figure 16. Similar 
behavior can be seen in comparison to the radar results (Figure 15). 
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Figure 16. AST Warning Alert Scoring for Combined Encounters 


Figure 17 shows results for the normalized distribution of Warning Alert scoring for the ADS-B- 
equipped runs. More than 80% of runs resulted in a Correct or Permissible Alert or Non-Alert, 
which are the most favorable alert scoring behaviors. 
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Figure 17. ADS-B Warning Alert Scoring for Combined Encounters 
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4.3 Alert Jitter -Open Loop 


To determine the effect of sensor uncertainty and the SUM uncertainty mitigation approach on 
alert jitter, open loop distributions of alert jitter were analyzed and are presented in Figures 18, 19, 
and 20 for radar, AST, and ADS-B, respectively. Each figure in this subsection shows alert jitter 
as a function of the data source provided to the DAA guidance system with Truth runs shown in 
blue, Sensed runs shown in red, and Mitigated runs shown in green. The Alert Jitter measure is the 
number of increasing alert transitions experienced during an encounter scenario. For an encounter 
in which both aircraft are in steady flight conditions, the ideal alert jitter value is “2.” An alert jitter 
value of “2” represents a transition from an initial alert state from <No Alert> to <Corrective 
Alert> to <Warning Alert>. High levels of alert jitter are considered a nuisance to the pilot in 
command, which may result in a lack of trust in the DAA system. There are no requirements 
associated with alert jitter presented in the DAA MOPS document, however consideration of this 
metric is important to the design of DAA systems. 


In Figure 18 the alert jitter distribution of radar-based runs is shown. There are a large number of 
Truth runs (i.e., runs with guidance based on perfect state data) with an alert jitter of “2,” as 
expected. There were a substantial number of Sensed runs with alert jitter of zero indicating that 
no alert was issued to the pilot-in-command (i.e., the pilot model) during the encounter, meaning 
the pilot model was given no cues indicating that there was a potential loss of separation. 
Compensating for sensor uncertainty using the SUM approach reduced the percentage of runs with 
no alert (alert jitter = 0) from 25% to 15%, while increasing the percentage of runs with the ideal 
alert jitter of “2” from 15% to 28%. Overall, the SUM approach is shown to change the sensed 
alert jitter distribution to be somewhat closer to the truth alert jitter distribution. 
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Figure 18. Open Loop Radar Alert Jitter for Combined Encounters 


Figure 19 shows the normalized distribution of alert jitter for AST-equipped simulation runs. As 
previously discussed, a large proportion of Truth runs resulted in an alert jitter of “2.” Relying on 
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AST as the only DAA sensor results in a wider range of alert jitter values. The SUM approach did 
not improve the distribution. It should be noted that the SUM approach was intended to address 
frequency and severity of losses of well clear, and a different approach to address alert jitter may 
be needed. 
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Figure 19. Open Loop AST Alert Jitter for Combined Encounters 


The normalized alert jitter distribution for runs with ADS-B equipped is presented in Figure 20. 
This figure shows a tightly bound alert jitter distribution not only for truth but also for both sensed 
and mitigated ADS-B data sources provided to the DAA system. Although the ADS-B system is 
able to provide more accurate sensed intruder position and velocity information to the DAA 
system, there are still occurrences of alert jitter greater than “7” for the Mitigated runs. These high 
values of alert jitter are associated with perturbations in the vertical rate of ‘phantom’ intruders. 
Further filtering of the vertical rate covariance term may help to reduce the alert jitter in these 
cases. 
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Figure 20. Open Loop ADS-B Alert Jitter for Combined Encounters 
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5. National Airspace System (NAS)-Derived Encounter Set Results 
5.1 Correlated Encounters 


5.1.1 Severity of Loss of Well Clear (SLOWC) 


Figure 21 shows a histogram of SLoWC for closed loop simulation for NAS-derived correlated 
encounters equipped with radar. The histogram also shows the open loop Truth data in navy blue 
color. Open loop Truth data shows the results with no pilot response in order to characterize the 
original encounter geometry and is compared to the closed loop Truth data shown in light blue. 
Results for these encounters show that the closed loop truth guidance resulted in 85% of encounters 
having a SLoWC value of 0%. 38% of closed loop Sensed radar runs resulted in a SLoWC value 
of 10% or less. The Mitigated source guidance increased the number of encounters with 0% 
SLoWC by 20% when compared to sensed only guidance. 
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Figure 21. Radar SLOWC for NAS-Derived Correlated Encounters 


Figure 22 shows a SLoWC histogram for AST guidance type. The SUM approach, enabled the 
DAA system to increase the number of runs with 0% SLoWC by approximately 26% more than 
Sensed guidance. 


Figure 23 shows a SLOWC histogram for ADS-B guidance type. Findings show that the Mitigated 


source guidance resulted in 8% more runs with 0% SLoWC than truth guidance, and 13% more 
runs than sensed guidance. 
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Figure 22. AST SLoWC for NAS-Derived Correlated Encounters 
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Figure 23. ADS-B SLoWC for NAS-Derived Correlated Encounters 
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5.1.2 Alert Scoring 


5.1.2.1 Corrective Alert Scoring 


Figure 24 shows a histogram of the corrective alert scoring for the radar sensor used in a closed 
loop simulation for NAS-derived correlated encounters. Results show that 85% of closed loop 
Truth runs fell within the “Permissible Alert” category, while 13% fell within the “Missed Alert” 
category. There were 56% fewer sensed guidance runs scored in the “Permissible Alert” than Truth 
guidance runs. The addition of the mitigated guidance resulted in 7% more runs scored in the 
“Permissible Alert” category than sensed guidance alone. 54% of runs using sensed guidance 
resulted in being scored in the “Missed Alert” category, however mitigated guidance resulted in 
39% of runs being scored in the “Missed Alert” category. Similar trends were seen for encounters 
using the AST sensor (Figure 25). Results for the ADS-B sensor (Figure 26) show that more than 
80% of closed loop encounters were within the “Permissible Alert” criteria signifying better 
performance by the ADS-B sensor. 
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Figure 24. Radar Corrective Alert Scoring for NAS-Derived Correlated Encounters 
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Figure 25. AST Corrective Alert Scoring for NAS-Derived Correlated Encounters 
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Figure 26. ADS-B Corrective Alert Scoring for NAS-Derived Correlated Encounters 
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5.1.2.2 Warning Alert Scoring 


A histogram of the warning alert scoring as a function of the radar sensor is shown in Figure 27. 
Results show that 22% of encounters in the closed loop truth guidance were scored as the 
“Permissible Alert” category and approximately 63% within the “Permissible Non-Alert” scoring 
category. No encounters were scored in the “Missed Alert” category using Truth guidance. There 
were 59% fewer sensed guidance encounters scored as a “Permissible Alert” when compared to 
truth guidance runs. The mitigated guidance resulted in 9% fewer runs scored in the “Missed Alert” 
category. 32% of encounters using Sensed guidance were within the “Missed Alert” scoring 
category, while the Mitigated guidance resulted in 23% of runs being scored in the “Missed Alert” 
category. The Mitigated guidance accounted for 10% more runs being scored in the “Permissible 
Alert” than the Sensed guidance runs. 


RADAR Warning Alert Scoring 
NAS-Derived Correlated Encounters 


40% @ Open Loop Truth 


20% & Closed Loop Truth 
0% _ I hk. L o m RADAR Closed Sensed 
cy Cy cy ee Cy Cy Cy Cy Cy ee mj RADAR Closed Mitigated 
oe ae oe ae ee oe OP ae ae 
é x De a se Se S 
x © AS) e < sO © ‘> od 
SP PF TK Fr KT VK 
eo & es N@ oe x ~\S 
= x & ‘> SS 
& 9 QV? & » QV 
s » 


Figure 27. Radar Warning Alert Scoring for NAS-Derived Correlated Encounters 


Result trends for encounters using the AST sensor (Figure 28) were similar to the radar sensor 
results. 


Results show encounters with Truth, Sensed, and Mitigated guidance had roughly similar results 
for the number of runs in the “Permissible Non-Alert” category (within 10% of each other) and 
were within 3% of each other in the “Permissible Alert” category. ADS-B showed better 
performance for warning alerts in comparison to the radar and AST sensors. 
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Figure 28. AST Warning Alert Scoring for NAS-Derived Correlated Encounters 
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Figure 29. ADS-B Warning Alert Scoring for NAS-Derived Correlated Encounters 
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5.1.3 Alert Jitter 


An alert jitter histogram is shown in Figure 30 as a function of the radar sensor used in a closed 
loop simulation for NAS-derived correlated encounters. Open loop results with Truth guidance 
show 74% of encounters with an alert jitter of “2”. One or two increasing alerts is ideal as it 
indicates a normal, steady progression in alert level without any alerts appearing and then 
disappearing. Closed loop Truth guidance shows 54% and 44% of encounters that experienced one 
and two alert increases, respectively. Mitigated results show that the SUM approach accounted for 
10% more runs with an Alert Jitter of “2” than sensed guidance runs. 
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Figure 30. Radar Alert Jitter for NAS-Derived Correlated Encounters 


Findings for encounters using the AST sensor (Figure 31) shows that Sensed and Mitigated 
guidance performed at comparable percentages among all the alert jitter values. 


Results for the ADS-B sensor (Figure 32) show that a high proportion of encounters had either an 
Alert Jitter of either “1” or “2”. Mitigated guidance with ADS-B shows the SUM approach had a 
less than 1% offset in comparison with the closed loop Truth guidance for the alert jitter value of 
“1.” However, it also spreads out the distribution to higher jitter values, which is undesirable. 
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Figure 31. AST Alert Jitter for NAS-Derived Correlated Encounters 
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Figure 32. ADS-B Alert Jitter for NAS-Derived Correlated Encounters 
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5.2. Uncorrelated Encounters 


5.2.1 Severity of Loss of Well Clear (SLOWC) 


Figure 33 shows a histogram of SLoWC for ownship using radar as the guidance source in a closed 
loop simulation in NAS-derived uncorrelated encounters. Results for these encounters show that 
the closed loop system resulted in 0% SLoWC for 55% of the encounters using Truth guidance. 
Whereas, using radar data for guidance with no uncertainty mitigation (Radar Closed Sensed) 
resulted in only 16% of the encounters achieving a 0% SLoWC. The SUM approach (Radar Closed 
Mitigated) enabled the DAA system to keep SLoWC at 0% approximately 50% more often than 
using Sensed guidance alone. 
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Figure 33. Radar SLoWC for NAS-Derived Uncorrelated Encounters 
Figure 34 shows a SLoWC histogram for ownship using AST as the guidance source in NAS- 
derived uncorrelated encounters. The SUM approach enabled the DAA system to keep SLoWC at 


0% approximately 28% more often than Truth guidance and approximately 43% more often than 
Sensed guidance. 


47 


AST SLoWC 
NAS-Derived Uncorrelated Encounters 


100% 


80% 
60% 

@ Open Loop Truth 
ae m Closed Loop Truth 
20% BAST Closed Sensed 

0% i es i re = AST Closed Mitigated 
oS 


a a a SR Ss a a 


Percentage of Encounters 


SLOWC % 


Figure 34. AST SLoWC for NAS-Derived Uncorrelated Encounters 


Figure 35 shows a SLoWC histogram for ownship using ADS-B data as the guidance source in 
NAS-derived uncorrelated encounters. Findings show that the Mitigated source guidance, as a 
result of the SUM model approach, kept SLoWC at 0% by approximately 22% more than Truth 
guidance and 26% more than Sensed guidance. 
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Figure 35. ADS-B SLoWC for NAS-Derived Uncorrelated Encounters 
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5.2.2 Alert Scoring 
5.2.2.1 Corrective Alert Scoring 


Figure 36 shows a histogram of the corrective alert scoring as a function of the radar sensor used 
in a closed loop simulation for NAS-derived uncorrelated encounters. Results from closed loop 
data runs using Truth guidance show almost all of the encounters scored in desirable categories 
(34% within the “Correct Required Alert,” 50% within the “Permissible Alert” criteria, and 5% 
for “Permissible Non-Alert’’) and relatively few scored in undesirable categories (5% for “Late 
Alert,” and 7% within the “Missed Alert” scoring criteria). The Sensed guidance dropped 
significantly in comparison to the Truth guidance for “Permissible Alert;” however, the Mitigated 
guidance was able to increase Permissible Alerts modestly. Mitigated guidance also enabled the 
DAA system to significantly decrease “missed alerts” at 21%. 
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Figure 36. Radar Corrective Alert Scoring for NAS-Derived Uncorrelated Encounters 


Figure 37 shows results seen for the AST sensor. Using Truth guidance, results show most closed 
loop encounters scored in desirable categories (28% within the “Correct Required Alert,” 53% 
within the “Permissible Alert” criteria, and 6% within the “Permissible non-Alert” criteria), and 
some scored in undesirable categories (8% within the “Missed Alert” and 5% within the “Later 
Alert” scoring criteria). Using Sensed guidance, the correct required and permissible alerts 
dropped significantly with a corresponding increase in short, incorrect, and missed alerts. The 
SUM approach was able to significantly decrease missed alerts, but the AST data is problematic 
enough that the uncertainty mitigation actually decreased the correct and permissible alerts and 
non-alerts but was able to increase the early permissible alerts. 
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Figure 37. AST Corrective Alert Scoring for NAS-Derived Uncorrelated Encounters 


Results for the ADS-B sensor are shown in Figure 38. Truth closed loop encounters scored within 
a 3% margin for the “Permissible Alert” criteria when compared to Sensed and Mitigated guidance. 
15% of encounters using Sensed guidance were missed alerts. However, the Mitigated guidance 
enabled the DAA system to match the Truth guidance results for that criteria. The ADS-B sensor 
performed better than the radar and AST sensors. 
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Figure 38. ADS-B Corrective Alert Scoring for NAS-Derived Uncorrelated Encounters 
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5.2.2.2 Warning Alert Scoring 


Figure 39 shows a radar sensor histogram of the warning alert scoring used in a closed loop 
simulation for NAS-derived uncorrelated encounters. Closed loop results show most encounters 
scored in desirable categories (34% within the “Correct Required Alert” and 55% for the 
“Permissible Alert” criteria), and 11% scored within the “Late Alert” criteria using Truth guidance. 
The Sensed guidance dropped precipitously in comparison to the Truth guidance for correct 
required and permissible alerts, with a corresponding increase in short and missed alerts. The 
Mitigated guidance was able to increase correct required alerts, correct non-alerts, and permissible 
alerts, and decrease short, late, and missed alerts. Unfortunately, using radar data it also increased 
incorrect warning alerts. 
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Figure 39. Radar Warning Alert Scoring for NAS-Derived Uncorrelated Encounters 


Figure 40 shows results seen for the AST sensor. Results show 90% of the closed loop encounters 
scored in desirable categories (32% within the “Correct Required Alert,” 58% within the 
“Permissible Alert” criteria), and 9% within the “Late Alert” scoring criteria using Truth guidance. 
The Sensed guidance decreased scores in desirable categories and increased scores in all 
undesirable categories. Mitigated guidance resulted decreases in short, late, and missed alerts, but 
also decreased correct required alerts and permissible alerts and non-alerts, and significantly 
increased incorrect alerts. The AST sensor performed least favorably in comparison to radar and 
ADS-B. 
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Figure 40. AST Warning Alert Scoring for NAS-Derived Uncorrelated Encounters 


Results for the ADS-B sensor are shown in Figure 41. For permissible alerts, Sensed scored within 
8% of Truth closed loop encounters and Mitigated scored within 7%. The “Correct Non-Alert” 
criterion contains 32% of encounters using Truth guidance, which is comparable to the 34% seen 
while using Sensed guidance. Mitigated guidance increased most desirable categories and 
decreased most undesirable categories. 
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Figure 41. ADS-B Warning Alert Scoring for NAS-Derived Uncorrelated Encounters 
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5.2.3 Alert Jitter 


Figures 42 through 44 show alert jitter performance of the DAA system using the three different 
sensors. For comparison, open loop results with no DAA system and no avoidance maneuvers are 
also shown alongside the closed loop results. In all three figures, just under 40% of the open loop 
with truth encounters scored an alert jitter of “2,” which is the desired number of increasing alerts, 
and the vast majority of the rest form a well-behaved tail out to 7. This tail indicates that in a 
significant number of these uncorrelated encounters one or both of the aircraft are maneuvering to 
the extent that DAAWC is lost or predicted to be lost more than once. Also, as can be seen in all 
three figures, using Truth data to guide the ownship only slightly increases the desired score of 2, 
and significantly increases a score of 4, indicating that very likely in most of those cases the 
ownship was able to maneuver to avoid LoWC or to regain DAAWC but the intruder subsequently 
maneuvered back into a LoWC situation. For reference, alert jitter is explained more in Section 
3.2.3 and at the beginning of Section 4.3. 


Figure 42 shows an alert jitter histogram for the radar sensor used in NAS-derived uncorrelated 
encounters in a closed loop simulation. Using the radar data for closed loop Sensed guidance pulled 
the distribution generally to the left, indicating the ownship maneuvered earlier and more 
drastically to avoid a LoWC because of the uncertainty in the data. Mitigated results show that the 
SUM approach was able to reshape the Sensed distribution to be closer to the Truth distributions. 
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Figure 42. Radar Alert Jitter for NAS-Derived Uncorrelated Encounters 
Results for the AST sensor (Figure 43) and ADS-B sensor (Figure 44) show that using unmitigated 
sensor data for guidance pulls the distribution generally to the right, indicating more maneuvering 


by the ownship. As with SLoWC, using the SUM approach worsened the AST alert jitter results, 
but seems to slightly flatten the distribution for ADS-B. 
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Figure 43. AST Alert Jitter for NAS-Derived Uncorrelated Encounters 
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Figure 44, ADS-B Alert Jitter for NAS-Derived Uncorrelated Encounters 
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6. MOPS Requirement-Derived Test Vectors Results 


As expected, overall performance of the MOPS-representative DAA system in both the Sensed 
and Mitigated cases was similar to the closed-loop Truth cases. For comparison, results of open 
loop runs — with no avoidance maneuvers — are also shown alongside the closed loop results. See 
Section 3.1.2 for an explanation of the MOPS requirement-derived test vectors. 


6.1 Severity of Loss of Well Clear (SLOWC) 


6.1.1 Radar SLOoWC 


Figures 45 through 48 show SLoWC histograms for the radar guidance source in a closed loop 
simulation using MOPS Requirement-Derived test vectors. 


Results for Head-On test vectors (Figure 45) show that in cases using Truth closed loop data for 
guidance, the ownship was always able to remain well clear from other aircraft (0% SLoWC). 
However, 1/3 of those same cases would have resulted in a mid-air collision (MAC) or near mid- 
air collision (NMAC) without the DAA system and pilot maneuvers (Open Loop Truth, 90-100% 
SLoWC). SLoWC performance for closed loop Sensed radar guidance was similar to closed loop 
Mitigated guidance, both of which are not far from performance for Truth data. 
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Figure 45. Radar SLoWC for Head-On MOPS Requirement-Derived Test Vectors 


Similarly, results for Converging test vectors (Figure 46) show that in cases using closed loop 
Truth data for guidance, the ownship was always able to remain well clear from other aircraft (0% 
SLoWC). One-third of those same cases, though, would have resulted ina MAC or NMAC without 
the DAA system and pilot maneuvers (Open Loop Truth, 90-100% SLoWC). Also, as with the 
Head-On Test Vectors, for the Converging Test Vectors performance with Sensed and Mitigated 
guidance spread out the distribution somewhat, slightly reducing the number of encounters 
remaining well clear (0% SLoWC) and slightly increasing those with values between 0 and 60. 
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Note that the DAA system was able to keep all Converging encounters using Sensed or Mitigated 
guidance below 60% SLoWC. 
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Figure 46. Radar SLOWC for Converging MOPS Requirement-Derived Test Vectors 


Results for Maneuvering test vectors (Figure 47) show that the ownship lost DAAWC in all 
encounters, regardless of the source of guidance data, even Truth. Performance remained 
consistent across all four cases (open Truth, closed Truth, Sensed, and Mitigated) with 50% of 
encounters having a SLoWC between 0% and 10%. The other half of the closed loop Truth 
encounters resulted in a 30-40% SLoWC. Note that the other half of both the Sensed and Mitigated 
encounters resulted in lower severity scores (10%-20%), likely because the sensor uncertainty 
caused the pilot model to maneuver earlier than when using Truth guidance. 
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Figure 47. Radar SLOWC for Maneuvering MOPS Requirement-Derived Test Vectors 


Figure 48 shows results for Overtake test vectors using radar data for guidance. The closed loop 
system resulted in 100% of encounters being within the 0% SLoWC range when Truth guidance 
was used. Open loop Truth results show 67% of encounters were within this SLoWC range and 
the remaining 33% of encounters were within the 20-30% SLoWC range. Overtake encounters 
using the radar sensor scored better in comparison to the other encounter geometries, which can 
be attributed to the radar’s field of regard specifications enabling it to always detect the intruder 
because the ownship is overtaking, or approaching, the other aircraft from behind. 
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Figure 48. Radar SLOWC for Overtake MOPS Requirement-Derived Test Vectors 
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6.1.2 Active Surveillance Transponder (AST) SLOoWC 


Figures 49 through 53 show SLoWC histograms for the AST guidance source used in a closed 
loop simulation for MOPS Requirement-Derived test vectors. 


Results for Head-On test vectors (Figure 49) show that the DAA system resulted in no losses of 
well clear for all encounters using Truth guidance. Sensed and Mitigated guidance encounters, 
however, were spread out across the entire range of SLoWC values, with performance similar to 
each other and to the open loop Truth results. 
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Figure 49. AST SLoWC for Head-On MOPS Requirement-Derived Test Vectors 


Similarly, results for Converging test vectors (Figure 50) show that the DAA system resulted in 
0% SLoWC for all encounters using Truth guidance. Sensed and Mitigated results were also spread 
out, but to only part of the SLoWC range. The SUM approach enabled the DAA system to pull the 
distribution to the left, eliminating all SLoWC values above 50% and increasing the number of 
well clear encounters to 79%. 
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Figure 50. AST SLoWC for Converging MOPS Requirement-Derived Test Vectors 


Figure 51 shows results for High Speed test vectors. Open loop Truth results show that all of these 
High Speed encounters would have some loss of well clear without any maneuvering to avoid it, 
including a significant portion (25%) having a SLOWC value close to 100%. Truth guidance in the 
closed loop system resulted in all encounters having 0% SLoWC. Both Sensed and Mitigated 
guidance spread the distribution out across essentially the entire range of SLoWC, but Mitigated 
guidance shows the SUM approach was able to keep 80% of the encounters well clear (SLoWC of 
0%), and increase of 17% over Sensed guidance. 
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Figure 51. AST SLoWC for High-Speed MOPS Requirement-Derived Test Vectors 
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Figure 52 shows results for Maneuvering test vectors using the AST sensor. Open loop Truth 
results show that all of the encounters would result in some loss of well clear, and the DAA system 
was able to prevent loss of well clear in only small portions of all maneuvering (closed loop) cases. 
However, the Mitigated approach had the highest percentage of encounters with no loss of well 
clear (33.86%). Overall, the DAA system performed least favorably in the Maneuvering 
encounters using the AST sensor in comparison to the other encounter geometries and sensors. 
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Figure 52. AST SLoWC for Maneuvering MOPS Requirement-Derived Test Vectors 


Figure 53 shows results for Overtake test vectors. Similar to the radar Overtake results, the closed 
loop system resulted in 100% of encounters having a 0% SLoWC when Truth guidance was used, 
but with some distribution over the SLoWC range with no maneuvers (Open Loop Truth). Both 
Sensed and Mitigated guidance results are very similar, avoiding loss of well clear in most 
encounters. 
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Figure 53. AST SLoWC for Overtake MOPS Requirement-Derived Test Vectors 
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6.1.3. Automatic Dependent Surveillance - Broadcast (ADS-B) SLOWC 


Figures 54 through 58 show SLoWC histograms for ADS-B guidance source used in a closed loop 
simulation for MOPS Requirement-Derived test vectors. ADS-B data is by far less noisy and 
uncertain than radar and AST; the figures show that the DAA system performs very close to the 
way it would with perfect (Truth) data. 


Results for Head-On test vectors (Figure 54) show that the DAA system resulted in no loss of well 
clear for all of the encounters using Truth guidance, whereas open loop Truth results show a 0% 
SLoWC value for only 33% of encounters. Essentially all the encounters using both Sensed and 
Mitigated guidance had a SLoWC of 0% (96% for closed loop Sensed and 97% for closed loop 
Mitigated. 
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Figure 54. ADS-B SLoWC for Head-On MOPS Requirement-Derived Test Vectors 


Results for Converging test vectors using ADS-B (Figure 55) are almost identical to the Head-On 
results. The DAA system was able to keep 99% of the encounters well clear using Sensed ADS-B 
guidance and 100% of encounters well clear using Mitigated guidance. 


62 


ADS-B SLOoWC 
Converging Test Vectors 
100% 
80% 


60% 
@ Open Loop Truth 


40% 
@ Closed Loop Truth 


20% | m@ ADS-B Closed Sensed 


» Wt 


. 5 PF PF fF F FC KF FS SF 
oso F PF F F F EF F FY 


Percentage of Encounters 


m ADS-B Closed Mitigated 


SLOWC % 


Figure 55. ADS-B SLoWC for Converging MOPS Requirement-Derived Test Vectors 


Figure 56 shows results for High Speed test vectors. Closed loop results show 100% of encounters 
remained well clear when any of the three guidance sources were used (Truth, Sensed, and 
Mitigated). 
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Figure 56. ADS-B SLOWC for High Speed MOPS Requirement-Derived Test Vectors 
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Figure 57 shows results for Maneuvering test vectors. As with AST and radar, the Maneuvering 
encounters for MOPS requirements testing proved very difficult, even for ADS-B. Only 13% of 
encounters had 0% SLoWC using Truth guidance and Sensed guidance results show only 12% of 
encounters maintained well clear; however, the Mitigated (SUM) approach more than doubled that 
(32% remained well clear). Overall, the DAA system performed least favorably in the 
Maneuvering encounters in comparison to the other MOPS requirements-derived encounter 
geometries. 
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Figure 57. ADS-B SLoWC for Maneuvering MOPS Requirement-Derived Test Vectors 


Figure 58 shows results for Overtake test vectors. The DAA system was able to keep 97% of 
encounters well clear using Sensed guidance and 100% of encounters well clear using Mitigated 
guidance. 
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Figure 58. ADS-B SLOWC for Overtake MOPS Requirement-Derived Test Vectors 
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6.2 Alert Scoring 


Traffic alerts and their scoring methodology are explained in Section 3.2.2. 


6.2.1 Corrective Alert Scoring 


Figures 59 through 72 show corrective alert scoring histograms for radar, AST, and ADS-B 
guidance sources used in a closed loop simulation for MOPS Requirement-Derived test vectors. 


6.2.1.1 Radar Corrective Alerts 


Figure 59 shows corrective alert results for Head-On test vectors using the radar sensor for closed 
loop testing. Closed loop Truth guidance shows 67% of encounters were within the “Correct Non- 
Alert” scoring criteria, which matches the open loop Truth results. The remaining 33% of closed 
loop Truth encounters were within the “Permissible Alert” scoring criteria. Using Sensed guidance 
decreased all desirable alerts and drastically increased incorrect alerts (45% of encounters). 
Mitigated guidance using the SUM approach worsened the situation, further decreasing desirable 
alerts and further increasing incorrect alerts (67% of encounters). 
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Figure 59. Radar Corrective Alert Scoring for Head-On MOPS Requirement-Derived Test 
Vectors 


Figure 60 shows results for Converging test vectors using the radar sensor. Closed loop Truth 
guidance shows 90% of encounters were within the “Permissible Alert” scoring criteria while the 
remaining 10% of encounters experienced permissible non-alerts. Using Sensed guidance 
decreases permissible alerts by 25%. Mitigated guidance using the SUM approach shows 57% of 
encounters received permissible alerts. Both Sensed and Mitigated shift a few of the encounters 
into the undesirable alert categories, with the mitigation approach making very little difference 
overall. 
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Figure 60. Radar Corrective Alert Scoring for Converging MOPS Requirement-Derived Test 
Vectors 


Findings from Maneuvering test vectors using the radar sensor are shown in Figure 61. Truth 
guidance in both open loop and closed loop shows 50% of encounters were within the “Correct 
Required Alert” scoring criteria while the remaining 50% never received a corrective alert. When 
Sensed and Mitigated guidance were used, no encounters received correct required alerts, nor any 
desired alerts. Sensed guidance shows 45% of test vectors encountered late alerts and 55% were 
within the “Missed Alert” scoring criteria. The SUM approach enabled the DAA system to alert 
on a few more encounters, but they were late. 
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Figure 61. Radar Corrective Alert Scoring for Maneuvering MOPS Requirement-Derived Test 
Vectors 
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Results for Overtake test vectors in a closed loop simulation and using the radar sensor are shown 
in Figure 62. Truth guidance shows 33% of encounters were within the “Correct Non-Alert” 
scoring criteria while the remaining 67% received permissible alerts. Using Sensed guidance 
decreased the number of encounters in both of those desirable categories, and shifted most of them 
to undesirable categories (22% incorrect alerts). Mitigated guidance made the alerting slightly 
worse (28% were within the “Incorrect Alert” scoring criteria). 
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Figure 62. Radar Corrective Alert Scoring for Overtake MOPS Requirement-Derived Test 
Vectors 
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6.2.1.2 Active Surveillance Transponder (AST) Corrective Alerts 


Figure 63 shows corrective alert results for Head-On test vectors using the AST sensor for closed 
loop testing. Truth guidance shows 21% of encounters were within the “Correct Non-Alert” 
scoring criteria while the remaining 79% were within the “Permissible Alert” scoring criteria. 
Surprisingly, using Sensed guidance shifts 25% of the encounters from Permissible Alerts to 
Correct Required Alerts, but also shifts encounters to Short (9%) and Missed Alerts (27%). 
Mitigated guidance using the SUM approach shows a decrease of 10% of the encounters that 
received missed alerts in comparison to Sensed guidance. 
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Figure 63. AST Corrective Alert Scoring for Head-On MOPS Requirement-Derived Test Vectors 


Figure 64 shows results for Converging test vectors using the AST sensor. Closed loop Truth 
guidance shows 92% of encounters were within the “Permissible Alert” scoring criteria while the 
remaining 8% of encounters received permissible non-alerts. Sensed guidance decreased 
permissible alerts 31% from Truth guidance, shifting about 15% to desirable correct required 
alerts, but also shifting the same number to short alerts and 9% to missed alerts. Mitigated 
guidance, using the SUM approach, further decreased permissible alerts (to 49%), it also decreased 
short and missed alerts. 
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Figure 64. AST Corrective Alert Scoring for Converging MOPS Requirement-Derived Test 
Vectors 


Figure 65 shows results for High Speed test vectors. Closed loop results of the Mitigated guidance 
using the SUM approach enabled 100% of the encounters to receive permissible alerts. Only 45% 
of encounters received permissible alerts using Truth guidance, while 63% of encounters were 
alerted in the same manner when Sensed guidance was used. 
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Figure 65. AST Corrective Alert Scoring for High Speed MOPS Requirement-Derived Test 
Vectors 
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Maneuvering test vectors results using the AST sensor are shown in Figure 66. Truth guidance in 
open and closed loop shows 57% of encounters received missed alerts while 63% of encounters 
received the same type of alert when Sensed Guidance was used. Mitigated guidance significantly 
decreased missed alerts to only 21%. The SUM approach also increased correct required alerts and 
early permissible alerts, with modest only increases in short, late, and incorrect alerts. 
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Figure 66. AST Corrective Alert Scoring for Maneuvering MOPS Requirement-Derived Test 
Vectors 
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Results for Overtake test vectors using AST are shown in Figure 67. All Truth guidance encounters 
scored in the desirable alert categories, with the majority (67%) in the “Permissible Alert” scoring 
criteria. Significantly fewer encounters (29%) using Sensed guidance received the same alert and 
fewer still (23%) when Mitigated guidance was used. Mitigated guidance shows the pilot model 
received early permissible alerts for 16% of encounters. Using Sensed guidance only 6% of 
encounters received early permissible alerts. Mitigated guidance shows 14% of encounters 
received missed alerts, which is 12% fewer encounters than when Sensed guidance was used. 
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Figure 67. AST Corrective Alert Scoring for Overtake MOPS Requirement-Derived Test Vectors 
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6.2.1.3 Automatic Dependent Surveillance — Broadcast (ADS-B) Corrective Alerts 


Figure 68 shows corrective alert results for Head-On test vectors using the ADS-B sensor. Results 
for the “Permissible Alert” scoring criteria show comparable findings within an 8% range when 
using closed loop Truth (80%), Sensed (72%), and Mitigated (78%) guidance. Comparable 
findings were also observed for the “Correct Non-Alert” scoring criteria (Open Loop Truth = 20%, 
Closed Loop Truth = 20%, and Sensed = 20%). Mitigated guidance results for correct non-alerts 
show 7% of encounters received that type of alert. 
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Figure 68. ADS-B Corrective Alert Scoring for Head-On MOPS Requirement-Derived Test 
Vectors 


Figure 69 shows results for Converging test vectors using the ADS-B sensor. Closed loop Truth 
guidance findings show 95% of encounters were within the “Permissible Alert” scoring criteria 
while 93% of encounters received the same alert when Sensed guidance was used and 91% when 
Mitigated guidance was used. Mitigated guidance findings show that 7% of encounters received 
early permissible alerts. Notably, none of the encounters using any guidance type scored in the 
undesirable categories. 
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Figure 69. ADS-B Corrective Alert Scoring for Converging MOPS Requirement-Derived Test 
Vectors 


Figure 70 shows corrective alert scoring results for High Speed test vectors as a function of ADS- 
B-based guidance. Open loop Truth results show 100% of encounters received correct required 
alerts, which indicates that an alert was issued for an encounter when an intruder aircraft was 
within the Hazard Zone. However, 100% of closed loop encounters received permissible alerts 
regardless of guidance used (Truth, Sensed, or Mitigated) indicating an alert was issued to the pilot 
model for an encounter when an intruder aircraft entered into the May-Alert Zone but before 
entering the Hazard Zone. Closed loop results may suggest that the pilot model received an alert 
ahead of the “Correct Required Alert” time seen in open loop runs in order to provide additional 
time for the ownship aircraft to maneuver. 
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Figure 70. ADS-B Corrective Alert Scoring for High Speed MOPS Requirement-Derived Test 
Vectors 


Figure 71 shows findings for Maneuvering test vectors using ADS-B-based guidance in closed 
loop testing. The majority of closed loop encounters (60%) using Truth guidance were observed 
within the “Missed Alert” scoring criteria, which compares directly to results seen when open loop 
testing was conducted. Sensed guidance results show slightly fewer (53% of encounters) received 
missed alerts. The SUM approach reduced the high percentage of missed alerts seen with Truth 
and Sensed guidance runs to 35%. 
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Figure 71. ADS-B Corrective Alert Scoring for Maneuvering MOPS Requirement-Derived Test 
Vectors 
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Corrective alert scoring results for Overtake test vectors using ADS-B are shown in Figure 72. 
Truth guidance runs show that the majority (69%) of closed loop encounters were within the 
“Permissible Alert” scoring criteria, with very comparable results for Sensed (63%) and Mitigated 
(64%) guidance. Open loop Truth shows the majority of encounters (67%) received correct 
required alerts while the remaining 33% received correct non-alerts. 
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Figure 72. ADS-B Corrective Alert Scoring for Overtake MOPS Requirement-Derived Test 
Vectors 


7D 


6.2.2 Warning Alert Scoring 


Figures 73 through 86 show warning alert scoring histograms for radar, AST, and ADS-B guidance 
sources used in a closed loop simulation for MOPS Requirement-Derived test vectors. 


6.2.2.1 Radar Warning Alerts 


Figure 73 shows warning alert results for Head-On test vectors using the radar sensor for closed 
loop testing. Truth guidance shows 33% of encounters were within the “Permissible Alert” scoring 
criteria while the remaining 67% of encounters experienced correct non-alerts. Sensed guidance 
reduced permissible alerts by 12% and correct non-alerts by 20%. For the “Incorrect Alert” scoring 
criteria, results show 19% of encounters were within that criteria when Sensed guidance was used; 
however, Mitigated guidance using the SUM approach greatly increased encounters receiving 
incorrect alerts to 67%. 
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Figure 73. Radar Warning Alert Scoring for Head-On MOPS Requirement-Derived Test Vectors 


Figure 74 shows warning alert results for Converging test vectors using the radar sensor. Closed 
loop Truth guidance shows 90% of encounters were within the “Permissible Alert” scoring criteria 
while the remaining 10% of encounters experienced permissible non-alerts. Sensed guidance 
shows a decrease of 25% in comparison to Truth guidance for permissible alerts, with the rest 
spread among several other criteria. Mitigated guidance using the SUM approach showed very 
little difference. 
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Figure 74. Radar Warning Alert Scoring for Converging MOPS Requirement-Derived Test 
Vectors 


Figure 75 shows results for Maneuvering test vectors using the radar sensor. Truth guidance shows 
100% of encounters were within the “Correct Required Alert” scoring criteria for both open and 
closed loop runs while only 1% of Sensed and 8% of Mitigated closed loop runs received those 
alerts, respectively. Sensed guidance shows 99% of test vectors encountered late alerts. Mitigated 
guidance runs were able to compensate for Sensed late alerts in only 7% of encounters. 
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Figure 75. Radar Warning Alert Scoring for Maneuvering MOPS Requirement-Derived Test 
Vectors 
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Results for Overtake test vectors using the radar sensor are shown in Figure 76. Truth guidance in 
closed loop runs show an even percentage of encounters (33%) were within the “Correct Non- 
Alert,” “Permissible Alert,” and the “Permissible Non-Alert” scoring criteria. Sensed guidance 
runs show 33% of encounters received correct non-alerts, 37% received permissible alerts, and 
21% received permissible non-alerts. The SUM approach shifted a significant number (40%) of 
encounters in these three desirable criteria to the “Early Permissible Alert” scoring criteria, but 
unfortunately shifted 45% of encounters to “Incorrect Alert.” 
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Figure 76. Radar Warning Alert Scoring for Overtake MOPS Requirement-Derived Test Vectors 
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6.2.2.2 Active Surveillance Transponder (AST) Warning Alerts 


Figure 77 shows warning alert results for Head-On test vectors for the AST sensor for closed loop 
testing. Truth guidance shows 43% of encounters were within the “Permissible Alert” scoring 
criteria, 29% within the “Permissible Non-Alert” criteria, and 29% within the “Correct Non-Alert” 
scoring criteria. As expected, sensed guidance decreases these three desirable alerts and increases 
short, late, incorrect, and missed alerts. However, it also increases correct required alerts; in 29% 
of these encounters the sensed position and velocity of the intruder aircraft led to a correct alert. 
The SUM approach increased desirable alerts and decreased undesirable alerts, with the notable 
exception of a significant increase in incorrect alerts. Incorrect alerts are essentially early alerts, in 
this case caused by the conservative nature of SUM’s approach to mitigating the sensor 
uncertainty. 
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Figure 77. AST Warning Alert Scoring for Head-On MOPS Requirement-Derived Test Vectors 


Results for warning alert scoring for AST for Converging test vectors are shown in Figure 78. 
Closed loop Truth guidance shows 92% of encounters were within the “Permissible Alert” scoring 
criteria while the remaining 8% were within the “Permissible Non-Alert” criteria. Sensed guidance 
shows 31% fewer in comparison to Truth guidance for permissible alerts, and the SUM approach 
reduced it even further to 49% of encounters. Although this percentage is less than that of Sensed 
(61%), Mitigated runs were also observed within the “Early Permissible Alert” scoring criteria for 
28% of encounters. For the “Short Alert” scoring criteria, results show the Mitigated guidance 
compensated for the Sensed guidance by decreasing the percentage of encounters within that 
criteria to only 4%. 
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Figure 78. AST Warning Alert Scoring for Converging MOPS Requirement-Derived Test Vectors 


Results for High Speed test vectors for warning alert scoring for AST are shown in Figure 79. The 
“Permissible Alert” scoring criteria shows 54% of encounters using Truth guidance were within 
that criteria while 23% of encounters were within the “Early Permissible Alert” criteria. Sensed 
guidance shows 63% of test vectors encountered permissible alerts. The SUM approach placed 
75% of encounters within the “Permissible Alert” scoring criteria with the remaining 25% of 
encounters within the “Permissible Non-Alert” scoring criteria. For this High Speed encounter set, 
the SUM approach performed exceptionally well, providing maneuver guidance such that no 
warning was needed at all in 25% of the encounters, and all others resulted in permissible alerts. 
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Figure 79. AST Warning Alert Scoring for High Speed MOPS Requirement-Derived Test Vectors 
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Results for Maneuvering test vectors for AST-based guidance are shown in Figure 80. The 
majority of closed loop encounters (71%) using Truth guidance were observed within the “Correct 
Required Alert” scoring criteria, which is 22% less than open loop Truth runs. Within that same 
scoring criteria, only 24% of encounters were observed using Sensed guidance and 27% when 
Mitigated guidance was used. 51% of Sensed guidance runs were within the “Late Alert” scoring 
criteria for maneuvering encounters; the SUM approach compensated for the Sensed guidance by 
decreasing the percentage of encounters within that scoring criteria to only 23%, which is 9% 
higher than the runs using Truth guidance. This set of maneuvering encounters is difficult for both 
Sensed and Mitigated guidance to handle, as evidenced by the significant distribution of 
encounters in the undesirable categories. 
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Figure 80. AST Warning Alert Scoring for Maneuvering MOPS Requirement-Derived Test 
Vectors 


Results for Overtake test vectors for AST are shown in Figure 81. Closed loop Truth guidance 
runs show results for 47% of encounters being within the “Permissible Alert” scoring criteria while 
27% of encounters were within the “Early Permissible Alert” criteria and the remaining encounters 
observed both correct required alerts and correct non-alerts. Sensed guidance shows 34% of test 
vectors received permissible alerts, and most of the rest received various undesirable alerts. The 
SUM approach correct required alerts and decreased most undesirable alerts, but also decreased 
permissible alerts and increased incorrect alerts. 
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Figure 81. AST Warning Alert Scoring for Overtake MOPS Requirement-Derived Test Vectors 


82 


6.2.2.3 Automatic Dependent Surveillance — Broadcast (ADS-B) Warning Alerts 


Figure 82 shows warning alert results for Head-On test vectors for the ADS-B sensor for closed 
loop testing. Results for the “Permissible Alert” scoring criteria show comparable findings when 
using closed loop Truth (40%), Sensed (41%), and Mitigated (38%) guidance. Comparable 
findings across the three guidance types were also observed for the “Permissible Non-Alert” 
scoring criteria (Truth = 33%, Sensed = 25%, and Mitigated = 33%). 
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Figure 82. ADS-B Warning Alert Scoring for Head-On MOPS Requirement-Derived Test 
Vectors 


Results for Converging test vectors for the ADS-B sensor are shown in Figure 83. Truth guidance 
findings in closed loop show 95% of encounters were within the “Permissible Alert” scoring 
criteria while 93% of encounters received the same alert when Sensed guidance was used and 91% 
when Mitigated guidance was used. Although Mitigated guidance shows a lesser percentage for 
permissible alerts when compared to Truth and Sensed guidance, findings show that 7% of 
encounters received early permissible alerts when the Mitigated guidance was used. This shows 
that the SUM approach slightly shifted encounters from within the “Permissible Alert” and 
“Permissible Non-Alert” scoring criteria when Sensed guidance was used by alerting the pilot 
model of an impending intruder aircraft at an earlier permissible time within the scenario. Warning 
alert scoring results for this type of encounter geometry are similar to those of the corrective alert 
scoring results that were presented in Figure 69. 
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Figure 83. ADS-B Warning Alert Scoring for Converging MOPS Requirement-Derived Test 
Vectors 


Figure 84 shows results for High Speed test vectors for the ADS-B sensor. Closed loop findings 
show that ADS-B sensor guidance performed better in High Speed encounters than in other 
encounter geometries when the ADS-B sensor was tested. Permissible alerts were received for 
78% of encounters when both Truth guidance was used. When Sensed guidance was used, 79% of 
encounters received similar alerts and 75% when Mitigated guidance was used. Mitigated guidance 
showed a slightly higher percentage of encounters (~3%) received permissible non-alerts in 
comparison to when Truth (22%) and Sensed (21%) guidance was used. 
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Figure 84. ADS-B Warning Alert Scoring for High Speed MOPS Requirement-Derived Test 
Vectors 
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Results for Maneuvering test vectors for ADS-B-based guidance are shown in Figure 85. The 
majority of closed loop encounters (73%) using Truth guidance were observed within the “Correct 
Required Alert” scoring criteria while 37% were observed using Sensed guidance and 47% when 
Mitigated guidance was used. Sensed guidance runs were predominantly (47%) within the “Late 
Alert” scoring criteria for maneuvering encounters; the SUM approach was able to decrease the 
percentage of encounters within that scoring criteria to only 20%. 
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Figure 85. ADS-B Warning Alert Scoring for Maneuvering MOPS Requirement-Derived Test 
Vectors 


Results for Overtake test vectors for ADS-B are shown in Figure 86. Truth guidance runs show 
44% of closed loop encounters being within the “Permissible Alert” scoring criteria while 31% of 
encounters were within the “Permissible Non-Alert” criteria and the remaining encounters 
received both correct required alerts and correct non-alerts. Sensed guidance shows 45% of test 
vectors received permissible alerts. The SUM model approach showed results for 44% of 
encounters within the “Permissible Alert” scoring criteria. 
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Figure 86. ADS-B Warning Alert Scoring for Overtake MOPS Requirement-Derived Test Vectors 
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6.3 Alert Jitter 


6.3.1 Radar Alert Jitter 


Figure 87 shows an alert jitter histogram for the radar sensor used for Head-On test vectors in a 
closed loop simulation. Open loop results with Truth guidance, which were evaluated against 
closed loop encounters, show 33% of encounters with an alert jitter of “0” indicating no alerts were 
given, which is undesirable. 33% of open loop Truth runs resulted in an alert jitter of “1” and the 
remaining third had an alert transition of “2,” which is the desirable number of increasing alerts. 
Closed loop Truth guidance matched results of open loop guidance for alert jitter of “0” and “1” 
but shows 0% of encounters experienced an alert jitter of “2.” Sensed guidance results were 
distributed across the spectrum; however, Mitigated results show that the SUM approach 
significantly tightened the distribution, to the extent that 65% of encounters received the ideal alert 
jitter of “2.” 
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Figure 87. Radar Alert Jitter for Head-On MOPS Requirement-Derived Test Vectors 


Figure 88 shows an alert jitter histogram for Converging test vectors in a closed loop simulation. 
Open loop results with Truth guidance show the majority of encounters (70%) presented the ideal 
number of increasing alerts, an alert jitter value of “2.” Closed loop results show only 20% of 
encounters had an alert jitter value of “2” using Truth guidance, and 16% of encounters when 
Sensed guidance was used. Mitigated results show that the SUM approach significantly improved 
the distribution of Sensed results, and increased encounters with an alert jitter value of “2” to 37%. 
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Figure 88. Radar Alert Jitter for Converging MOPS Requirement-Derived Test Vectors 


An alert jitter histogram for Maneuvering test vectors is shown in Figure 89 for radar-based 
guidance. Open loop and closed loop results with Truth guidance show 100% of encounters 
presented the ideal number of increasing alerts, which is a value of “2” alerts per encounter. Closed 
loop Sensed guidance shows only 24% of encounters had an alert value of “2,” and only 3% of 
encounters when Mitigated guidance was used. Mitigated results show that, although the SUM 
approach did eliminate one part of the tail of the Sensed distribution, it also shifted a significant 
portion of Sensed encounters with an alert jitter value of “2” to a jitter value of “1.” 
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Figure 89. Radar Alert Jitter for Maneuvering MOPS Requirement-Derived Test Vectors 
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Alert jitter results for Overtake test vectors are shown in Figure 90 for radar-based guidance. Open 
loop results with Truth guidance show the majority of encounters (67%) received two alert 
transitions while the remaining 33% received only one alert transition. Closed loop Truth guidance 
shows 33% of encounters received an alert jitter of “2” while 67% received one alert transition. 
As expected, Sensed guidance results show a fairly wide distribution of alert jitter values. 
However, Mitigated results show that the SUM model approach somewhat increased the number 
of encounters with an alert jitter value of “2,” but had very little effect on the distribution. 


RADAR Alert Jitter 
Overtake Test Vectors 


@ Open Loop Truth 
m Closed Loop Truth 
m RADAR Closed Sensed 


20% | | m@ RADAR Closed Mitigated 
0% I L a = = 
2 3 4 5 6 


7 8 9+ 


Percentage of Encounters 


Number of Increasing Alerts 


Figure 90. Radar Alert Jitter for Overtake MOPS Requirement-Derived Test Vectors 
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6.3.2 Active Surveillance Transponder (AST) Alert Jitter 


Figure 91 shows an alert jitter histogram for the AST sensor used for Head-On test vectors in a 
closed loop simulation. Open loop results with Truth guidance show 50% of encounters received 
the desired two alert transitions. Closed loop Truth runs, however, only show 14% of encounters 
received two alert transitions while the majority of encounters received either one (36%) or three 
(36%) alert transitions. Sensed and Mitigated results show sporadic findings across all transition 
alert levels (0O—9+); however, the SUM approach shifted the distribution slightly to the right. Across 
all alert jitter results, the DAA system performed least favorably using AST-based guidance. 
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Figure 91. AST Alert Jitter for Head-On MOPS Requirement-Derived Test Vectors 


Figure 92 shows an alert jitter histogram for Converging test vectors in a closed loop simulation. 
Open loop results with Truth guidance show the majority of encounters (77%) received the ideal 
number of alert transitions, “2.” Closed loop Truth guidance shows only 23% of encounters had 
an alert jitter value of “2” while the majority of encounters (54%) received one alert transition. 
Sensed and Mitigated results show sporadic findings across all transition alert levels (0Q—9+). 
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Figure 92. AST Alert Jitter for Converging MOPS Requirement-Derived Test Vectors 


Alert jitter results for High Speed test vectors are shown in Figure 93 for AST-based guidance. 
Open loop results with Truth guidance show 100% of encounters received two alert transitions 
while 75% of closed loop Truth runs received two alert transitions. Findings for Sensed and 
Mitigated guidance runs show only 19% of encounters received the ideal number of alert jitter 
while the remaining encounters using Sensed and Mitigated guidance showed sporadic behavior 
across the remaining transition alert levels. 
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Figure 93. AST Alert Jitter for High Speed MOPS Requirement-Derived Test Vectors 
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An alert jitter histogram for Maneuvering test vectors is shown in Figure 94. Open loop and closed 
loop results with Truth guidance show 57% of encounters presented the ideal number of increasing 
alerts, which is a value of “2” alerts per encounter, while the remaining 43% received one alert 
transition. Sensed and Mitigated guidance showed sporadic behavior across the presented 
transition alert levels (0O—9+). 
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Figure 94, AST Alert Jitter for Maneuvering MOPS Requirement-Derived Test Vectors 


Alert jitter results for Overtake test vectors are shown in Figure 95 for AST-based guidance. Open 
loop results with Truth guidance show 40% of encounters received two alert transitions, 20% 
received one alert transition, and 33% received three alert transitions. Closed loop Truth guidance 
shows 33% of encounters received an alert jitter of “2” and another 33% of encounters received 
one alert transition. Mitigated results show sporadic findings across presented transition alert levels 
(0—9+). In alert jitter, AST-based guidance caused the DAA system to perform least favorably 
among other sensor-based guidance. 
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Figure 95. AST Alert Jitter for Overtake MOPS Requirement-Derived Test Vectors 
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6.3.3. Automatic Dependent Surveillance - Broadcast (ADS-B) Alert Jitter 


Figure 96 shows an alert jitter histogram for Head-On test vectors for the ADS-B sensor used in a 
closed loop simulation. Open loop results with Truth guidance show 53% of encounters received 
two alert transitions while the remaining encounters received either zero alerts (7%), one alert 
transition (13%), or three alert transitions (27%). Closed loop Truth guidance shows 13% of 
encounters experienced an alert jitter of “2” which dropped to 27% of encounters when Sensed 
guidance was used. Mitigated results, however, show that the SUM approach somewhat tightened 
the Sensed distribution, enabling 39% of encounters to receive the ideal number, “2,” of alert 
transitions. 
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Figure 96. ADS-B Alert Jitter for Head-On MOPS Requirement-Derived Test Vectors 


Figure 97 shows an alert jitter histogram for Converging test vectors in a closed loop simulation. 
Results for open loop runs using Truth guidance show the majority of encounters (80%) received 
two alert increases. Closed loop Truth guidance shows only 25% of encounters received two, while 
the majority of encounters (60%) received one alert transition. For runs using Sensed guidance, 
results show a distribution somewhat similar to the Truth distribution for the first four jitter values, 
but with a much longer tail. Mitigated guidance tightens the distribution only slightly and does not 
eliminate the significant tail. 
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Figure 97. ADS-B Alert Jitter for Converging MOPS Requirement-Derived Test Vectors 


Alert jitter results for High Speed test vectors are shown in Figure 98 for ADS-B-based guidance. 
Open loop Truth guidance results 100% of encounters received two alert transitions while 75% of 
closed loop Truth runs received an alert jitter of “2.” 61% of encounters received two alert 
transitions when Sensed guidance was used, with very few greater than two. The SUM approach 
tightened the distribution slightly. 
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Figure 98. ADS-B Alert Jitter for High Speed MOPS Requirement-Derived Test Vectors 


Alert jitter results for Maneuvering test vectors are shown in Figure 99 for ADS-B-based guidance. 
Results for open loop and closed loop runs using Truth guidance show 53% of encounters received 
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two alert transitions whiles 47% of encounters received one alert jitter. Closed loop Sensed 
guidance shows 47% of encounters had an alert jitter value of “2” and 35% had an alert jitter value 
of “1.” The SUM approach actually worsened the situation, reducing the number of encounters 
with one, two, and three alert increases and causing four or more alert increases in over a third of 
the Maneuvering Test Vector encounters. 
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Figure 99. ADS-B Alert Jitter for Maneuvering MOPS Requirement-Derived Test Vectors 


Alert jitter results for Overtake test vectors are shown in Figure 100 for ADS-B-based guidance. 
Open loop results with Truth guidance show a small distribution of encounters around an alert 
jitter value of “2” (44% received two alert transitions, 31% received three alert transitions, 19% 
received one alert transition and 6% received no alerts). Closed loop Truth results show a widening 
distribution, with fewer encounters (31%) receiving two alert transitions and more (38%) receiving 
one alert transition. Sensed results are similar to Truth, except with a significant tail. Mitigated 
results show comparable findings to closed loop Truth and Sensed. 
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Figure 100. ADS-B Alert Jitter for Overtake MOPS Requirement-Derived Test Vectors 
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6.4 “Off-Nominal” Cases 


In order to determine the causes behind the encounters in E2-V2 where high max SLoWC was 
measured, a few of the most severe cases were investigated individually. Cases with SLoWC above 
70% which were also determined to have entered the NMAC volume at Closest Point of Approach 
(CPA) were selected for further analysis from the overall pool of closed-loop encounters. Replay 
animations and time history charts were used to analyze each case. Each high SLoWC case was 
compared to its respective truth surveillance run to determine whether inconsistent or incorrect 
guidance caused the pilot model to maneuver towards the intruder or if a LoWC was physically 
unavoidable given the encounter geometry. Three overall causes were uncovered in the case 
studies: maneuvering intruders which were impossible to avoid, the unrealistic behavior of the 
pilot model, and the guidance which resulted from the sensor uncertainty mitigation under certain 
conditions. 


Maneuvering encounter 304 demonstrates a case where the blunder maneuver of the intruder 
creates a situation where a LoWC was unavoidable. The intruder aircraft begins the encounter 
laterally offset by 2500 feet traveling at 600 KTAS and the ownship traveling in the opposite 
direction at 250 KTAS. At a range of approximately 3 nmi, the intruder aircraft initiates an abrupt 
90° turn towards the ownship. Figure 101 illustrates the geometry of encounter 304 with the closed 
loop ownship track histories for each run. Overall, the ownship response to the maneuvering 
intruder was uniform with all runs making an initial right turn away from the intruder. When 
provided with truth data (black line in Figure 101) the ownship makes an initial right turn to avoid 
the intruder while it was on an unchanging course followed by a more drastic ownship maneuver 
when the intruder began its turn towards the ownship. Many of the mitigated runs (red lines in 
Figure 101) also appear to follow this pattern of making an initial maneuver followed by a larger 
avoidance maneuver. 
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Figure 101. Track History for Encounter 304 
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The SLoWC histogram of all runs for encounter 304 (Figure 102) demonstrates that a LoWC was 
unavoidable for encounter 304. The severity of loss of well clear for all encounters ranged from 
47% to 99% maximum SLoWC. The sensor uncertainty mitigation appeared to reduce the SLloWC 
compared to the truth and sensed runs, primarily due to the magnitude and timing of the initial 
ownship turn before the intruder’s blunder which increased separation between the ownship and 
intruder. 


Maneuvering Encounter 304 SLOWC ADS-B 


Figure 102. Histogram of all runs for Encounter 304 


Figure 103 is a heading, alerting, and guidance time history of encounter 304 using truth 
surveillance data. The X-axis represents time in seconds since the start of the encounter and the 
Y-axis represents heading deviation of the ownship from the start heading. The current heading of 
the ownship is represented as a gray heading if no DAA alert is present, yellow if corrective alert 
is present and red if a warning alert is present. The black dots represent the heading commanded 
by the pilot model. The position of the DAIDALUS avoidance bands relative to the initial heading 
for each time step in the simulation is represented as orange bars and the recovery bands as green 
bars. Estimated time to LoWC at each time step (read in seconds on the Y-axis) is represented by 
purple triangles; purple triangles at the zero line represents a measured LoWC. The encounter 
begins benignly with a correct required corrective alert and a commanded heading approximately 
10° from the start heading. The alerting changes back to the no alert level with the estimated time 
to LoWC reaching approximately 30 seconds. At 190 seconds after the beginning of the scenario 
the avoidance bands abruptly grow and a late warning alert was logged with a LOWC and NMAC 
subsequently logged. The pilot model eventually commanded a 30° heading change after the 
LoWC and returned to start heading after a LOWC was no longer projected by DAIDALUS. The 
maximum SLoWC measured in this run was 92% with an eventual CPA of 358 feet. The 
simultaneous warning alert and LoWC, with no immediately preceding alerting and guidance, 
using truth data, essentially means a LoWC was unavoidable. It is conceivable that a large 
avoidance maneuver very early in the encounter may have avoided a loss of well clear, but alerting 
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and guidance occurring that early would require an unacceptably significant expansion of the well 
clear volume. 
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Figure 103. Alerting and Guidance Time History for Encounter 304 with Truth Surveillance 
Data 


Figure 104 shows encounter 304 using sensed ADS-B surveillance data and displays a pattern 
similar to the truth surveillance data run. A correct required corrective alert was displayed and the 
pilot model commands a heading change of approximately 10° before a correct required warning 
alert was triggered. The estimated time to LoWC reaches approximately 23 seconds before the 
aircraft has maneuvered enough for the alerting to return to the no alert level. At 190 seconds after 
the start of the encounter a late DAA warning was triggered and a LoWC was subsequently logged. 
The pilot model immediately commanded a single heading change to approximately 20° before 
alerting and guidance no longer indicated a projected LoWC. The maximum SLoWC measured 
was 99.5% with an eventual CPA of 42 feet. Similar to the truth run for this encounter the 
simultaneous warning alert and LoWC indicate the abrupt blunder maneuver by the intruder 
aircraft made a LoWC unavoidable, however the smaller second commanded maneuver by the 
ownship in the sensed run apparently increased the severity of well clear loss. 
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Figure 104. Alerting and Guidance Time History for Encounter 304 Sensed ADS-B Surveillance 
Senor 


Figure 105 shows the alerting and guidance time history for encounter 304 using a mitigated ADS- 
B track. Once again a correct required corrective alert was displayed and a 25° heading deviation 
was commanded by the pilot model. A correct required warning alert was displayed before the 
alerting returned to the no alert level. A late warning alert and immediate LoWC were triggered 
by the maneuvering intruder, similar to the sensed run. The pilot model commanded an additional 
maneuver of about 40° before the alerting and guidance no longer indicated a projected LoWC and 
commanded a return to the start heading. The maximum SLoWC for this run was 48% with an 
eventual CPA of 1996 feet. In this case the sensor uncertainty mitigation caused the pilot model to 
make a larger initial deviation than was commanded in the truth or sensed runs, minimizing the 
max SLoWC and maximizing the CPA. 
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Figure 105. Alerting and Guidance Time History for Encounter 304 with Mitigated ADS-B 
Surveillance Sensor 


The two other causes for high SLoWC encounters included unrealistic pilot model behavior and 
unusual guidance caused by the sensor uncertainty and mitigation. In cases where the sensed flight 
state is noisy or the guidance is frequently changing the favored direction of ownship maneuver, 
the pilot model can change the commanded heading from one direction to another. This can result 
in the ownship performing an initial correct deviation away from the intruder but then turn back 
towards the intruder or vice versa because of erroneous state information input to DAIDALUS. 
While it is possible a human UAS operator would follow guidance indicating a favored turn 
towards an oncoming intruder, it is more likely that a human operator would continue or increase 
an initial deviation. In cases where SUM is utilized and sensed flight state uncertainty is high, it is 
possible that the phantom intruders’ future predicted flight states diverge to the point where the 
DAIDALUS algorithm can find well clear airspace between the phantom intruder as multiple 
aircraft. The net result of this in the DAA guidance was multiple, segmented bands which can lead 
the pilot model to maneuver between bands only for them to merge into a single band when the 
intruder approaches close enough. 


Head-on encounter 216 using the AST sensor provides examples of both unrealistic pilot model 
behavior and sensor uncertainty mitigation which caused higher SLoWC values in some cases. 
The encounter begins with the intruder at a 3600 feet lateral offset at 600 KTAS while the ownship 
is at 200 KTAS flying in the opposite direction. The intruder was in a 1500 ft/min descent while 
the ownship was climbing at 1000 ft/min. Figure 106 shows that the track histories of the truth 
(black line), sensed (blue lines) and mitigated runs (red lines) are widely scattered between making 
left and right deviations, indicating the AST sensor produced multiple, scattered, intruder tracks 
under identical (Truth) circumstances. 
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Figure 106. Track History for Encounter 216 with AST Surveillance Sensor 


The SLoWC histogram of all runs for encounter 216 (Figure 107) shows that a large proportion of 
the sensed and mitigated runs avoided a LoWC altogether, indicating that LoWC occurrences were 
avoidable. Although the distribution of maximum SLoWC appears to be skewed towards low 
values or no LoWC, there were a significant number of runs above 70%. 
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Figure 107. Maximum SLoWC Histogram for Encounter 216 with AST Surveillance Sensor 
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Figure 108 shows the alerting and guidance time history for encounter 216 using truth data. A 
permissible corrective alert was triggered followed by a single deviation commanded by the pilot 
model and alerting subsequently stopped indicating a loss of well clear; it was no longer predicted. 
A LoWC was not logged in this run and the CPA was well outside the lateral distance alerting 
threshold at 8209 feet. 
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Figure 108. Alerting and Guidance Time History for Encounter 216 with Truth Surveillance 
Data 


Figure 109 shows the alerting and guidance time history for encounter 216 using the AST 
surveillance sensor. The encounter begins with guidance bands off to one side of the ownship 
aircraft’s nose which shift to cause a correct required corrective alert. The corrective alert was 
extinguished and almost immediately triggered again before being extinguished again. A correct 
required warning alert was triggered and the pilot model immediately commands a 15° deviation 
followed approximately 10 seconds later by a commanded 65° deviation following the changing 
band guidance. Following this, the recovery guidance bands briefly indicate a turn in the opposite 
direction was desired and the pilot model commanded an 89° turn in the opposite direction as a 
response. Three seconds after the change in direction was commanded, a LoWC was logged. The 
ownship began to change directions to follow the commanded heading, however the guidance 
bands once again reversed and the pilot model commanded a heading in the original direction. The 
ownship was eventually projected to be well clear of the intruder at 209 seconds after the start of 
the encounter. The maximum measured SLOWC was 78% and the eventual CPA was 847 feet. 
Although it is possible the ownship could have still lost well clear if the course reversal had not 
taken place, the reversal almost certainly increased the severity of well clear loss. 
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Figure 109. Alerting and Guidance Time History for Encounter 216 with Sensed AST 
Surveillance Sensor 


Figure 110 shows the alerting and guidance time history for encounter 216 using the sensor 
uncertainty mitigation with the AST surveillance sensor. The effect of the high sensor uncertainty 
coupled with the uncertainty mitigation and high intruder speed caused segmented bands to be 
displayed starting at 108 seconds after the start of the encounter. This was a direct result of the 
DAIDALUS algorithm sensing the multiple phantom intruders being generated by the SUM as 
their projected flight states diverge. This caused the alerting to increase levels 6 times throughout 
the encounter and the commanded heading to change 7 times, with 2 complete commanded 
heading direction reversals. The measured maximum SLoWC was 93% and the eventual CPA was 
237 feet. Future improvements to the SUM should prevent this issue by forcing the algorithm to 
sense only one phantom intruder. 
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Figure 110. Alerting and Guidance Time History for Encounter 216 with Mitigated AST 
Surveillance Sensor 
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7. Conclusion 


The demand for unmanned aircraft in mainstream aviation operations continues to grow. 
Understanding key detect and avoid system performance capabilities and limitations are essential 
to developing rules and regulations that allow routine UAS operations but maintain the safety of 
the National Airspace System. To understand these capabilities and limitations, as part of on-going 
RTCA SC228 efforts, NASA Langley Research Center evaluated the Phase I DAA MOPS 
requirements with end-to-end functionality over a specific set of encounters, in a variety of 
geometries, and with specific surveillance sensor performance, in order to verify and validate that 
a MOPS-representative DAA system performs acceptably. 


Evaluation results showed that overall, a MOPS-representative DAA system performed within 
acceptable ranges with few limitations. Values greater than 50% for severity of loss of well clear 
(SLoWC) occurred in less than 1.5% of total encounters. Results for alert scoring and alert jitter 
were similar. Losses of well clear and poor alert performance were mainly due to late maneuvers 
made by the intruder aircraft that the DAA system could not guard against and shortcomings of 
the surveillance data available to the DAA system. In particular, the AST sensor produced very 
inaccurate and noisy intruder tracks that caused multiple issues with SLoWC and alerting, and 
experienced data dropouts in about 70% of the MOPS requirements-derived test vector runs. 
Results suggest that slow moving aircraft should not depend solely on the AST sensor for lateral 
maneuvers. It should be noted that all three surveillance sensors (ADS-B, Phase I air-to-air radar, 
and AST) were modeled to produce data at the minimum specified quality, and can be expected to 
perform better in the field on average. As expected, the DAA system performed better with ADS-B 
than with radar or AST. Two other factors affecting DAA system behavior were observed in the 
pilot model. In some cases, the update delays in the pilot model were longer than a human operator 
would likely have taken, and the current direction of maneuver was not considered, which led to 
some cases in which the pilot model tried to steer the UA out of an avoidance maneuver that a 
human operator would have simply maintained or increased. Also, in some encounters with high 
uncertainty about the intruder’s position and velocity, the sensor uncertainty mitigation (SUM) 
approach used in this study could cause erroneous guidance and alerting. Taken all together, none 
of the results of this study revealed surprising or serious problems with a Phase I DAA MOPS- 
compliant system. 
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Appendix A - Depiction of Encounter Geometries 
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Figure A 1. Notional Diagram of Head-On Encounter Geometry 
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Crossing Encounter Geometry 
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Figure A 2. Notional Diagram of Crossing Encounter Geometry 
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Overtake Encounter Geometry 


anaceana} 


Figure A 3. Notional Diagram of Overtake Encounter Geometry 
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